From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning

Xitai Jiang; Zihan Tang; Wenze Lin; Yang Yue; Shenzhi Wang; Gao Huang

Source-linked AI summary

From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning

Xitai Jiang, Zihan Tang, Wenze Lin, Yang Yue, Shenzhi Wang, Gao Huang

arXiv:2605.22074v1May 21, 2026cs.LGcs.AIcs.CL

TL;DR

Outcome-based RLVR provides sparse feedback on hard reasoning problems and cannot adequately credit partial progress in failed rollouts. SCRL constructs verifiable subproblem curricula from reference solutions and assigns subproblem-level advantages to answer spans during on-policy training. Across seven mathematical benchmarks, it outperforms strong baselines, with larger relative gains as problem difficulty increases.

Problem
Outcome-based RLVR is inefficient on hard problems because correct final-answer rollouts are rare and sample-level credit assignment cannot use partial progress in failed attempts.
Method
SCRL derives easier-to-harder verifiable subproblems from reference reasoning chains, fixes the final subproblem as the original problem, and assigns normalized subproblem rewards to corresponding answer spans.
Results
+4.1/+1.9 average-point gains over GRPO on Qwen3-4B/14B are reported across seven mathematical reasoning benchmarks.
Takeaways & Limitations
Subproblem curricula lift hard problems out of gradient dead zones, with larger relative gains as the original problem becomes harder.
Takeaways & Limitations
SCRL assumes each subproblem has an objectively checkable answer and its theory includes bounded intermediate success probabilities for j < K.

Abstract

from arXiv · show

Reinforcement learning from verifiable rewards (RLVR) has shown strong promise for LLM reasoning, but outcome-based RLVR remains inefficient on hard problems because correct final-answer rollouts are rare and sample-level credit assignment cannot use partial progress in failed attempts. We introduce SCRL (Subproblem Curriculum Reinforcement Learning), a curriculum RL framework that derives verifiable subproblems from reference reasoning chains and fixes the final subproblem as the original problem. This turns partial progress on hard problems into verifiable learning signals. Algorithmically, SCRL uses subproblem-level normalization, which normalizes rewards independently at each subproblem position and assigns the resulting advantages to the corresponding answer spans, enabling finer-grained credit assignment without external rubrics or reward models. Our analysis shows that subproblem curricula lift hard problems out of gradient dead zones, with larger relative gains as the original problem becomes harder. Across seven mathematical reasoning benchmarks, SCRL outperforms strong curriculum-learning baselines, improving average accuracy over GRPO by +4.1 points on Qwen3-4B-Base and +1.9 points on Qwen3-14B-Base. On AIME24, AIME25, and IMO-Bench, SCRL further improves pass@1 by +3.7 points and pass@64 by +4.6 points on Qwen3-4B-Base, indicating better exploration on hard reasoning problems.

1. Introduction

SCRL addresses sparse rewards and coarse credit assignment in hard-problem RLVR by turning reference reasoning chains into verifiable subproblem curricula. It combines subproblem-level credit assignment with on-policy exploration and shows consistent theoretical and empirical gains.

Motivation: Hard problems expose outcome-based RLVR to sparse final-answer rewards and weak credit assignment for partial progress.Correct final answers are automatically verifiable, but failed rollouts provide limited learning signal.
SCRL framework: SCRL derives an easier-to-harder sequence of verifiable subproblems from a reference solution while fixing the final subproblem as the original problem.The model answers all subproblems in a single on-policy rollout, with later parts building on earlier results.
Analysis and validation: Theoretical analysis shows subproblem decomposition lifts hard problems out of gradient dead zones, with larger relative gains as difficulty increases.Experiments across benchmarks are consistent with this prediction, and ablations support the credit-assignment design.
Curriculum learning: SCRL keeps exploration near the model’s capability boundary by making intermediate progress on hard problems verifiable within on-policy rollouts.The resulting curriculum lets earlier solved subproblems guide increasingly difficult reasoning steps.
Credit assignment: Subproblem-level normalization independently normalizes rewards by subproblem position and assigns advantages to corresponding answer spans.This provides token-level, process-aware credit assignment without external rubrics, process annotations, or additional reward models.

2. Related Work

Prior curriculum methods ease mathematical reasoning through external hints, rationales, or rewritten prompts. SCRL instead requires the model to generate the scaffolded multi-part sequence itself, addressing a shared reliance on static auxiliary context.

RLVR: RLVR trains reasoning models with automatically checkable rewards on deterministic tasks such as mathematics and programming.This distinguishes RLVR from open-ended generation and supports policy optimization with PPO or group-based methods.
Existing curriculum methods: One curriculum paradigm provides external hints or teacher-generated rationales to lower the exploration threshold on challenging problems.These methods supply auxiliary prefixes when the model fails or needs guidance.
Existing curriculum methods: Another paradigm rewrites problems or augments prompts with supplementary information to create a difficulty gradient.Examples manipulate problem context rather than requiring the model to construct the scaffold itself.
SCRL distinction: Static hints and reformulated prompts primarily optimize continuation capability because the model is not required to generate the auxiliary structures.SCRL contrasts with this design by requiring the model to produce the entire scaffolded multi-part sequence within its rollout.

3. Method

SCRL constructs verifiable subproblem curricula from reference solutions, has the policy solve them in one structured rollout, and converts progress into span-level learning signals. Mixed-group training combines these curriculum rollouts with standard original-problem GRPO.

Curriculum construction: SCRL uses an external LLM to derive K verifiable subproblems from a reference solution, then verifies the policy’s answers during one on-policy rollout.The curriculum is built from intermediate progress nodes rather than solving the original problem from scratch.
Curriculum construction: Subproblems are ordered from easier to harder, linked through intermediate results, independently self-contained, and finalized with the original problem.Each subproblem has an objectively checkable answer, while later parts build on earlier ones.
Token-level credit assignment: Tagged spans <pj>...</pj> identify each subproblem answer, allowing separate verification and assignment of its advantage to the corresponding tokens.Tokens outside answer spans receive zero advantage.
Progress-aware rewards: Curriculum progress records the maximum consecutively solved prefix, with k_i = 0 meaning the first subproblem fails and k_i = K meaning all are solved.Progress-aware correction removes rewards for later subproblems after an earlier failure; for example, (1, 1, 0, 1) becomes (1, 1, 0, 0).
Subproblem-level normalization: Subproblem-level normalization computes advantages independently at each position, measuring each rollout’s relative success without using rewards from other positions.These position-specific advantages are then used for fine-grained credit assignment.
Mixed-group training: Mixed-group training uses half curriculum rollouts with subproblem-level advantages and half original-problem rollouts with standard outcome-based GRPO.This addresses prompt mismatch between curriculum training and original-prompt evaluation.

4. Theoretical Analysis

The analysis characterizes outcome-based GRPO as vulnerable to gradient dead zones on hard problems and models subproblem decomposition as metric recovery. The recovery becomes relatively larger as the original problem becomes harder, under stated subproblem and identifiability assumptions.

Metric recovery: SCRL lifts optimization from the original policy manifold to a subproblem product manifold, recovering useful gradient information from intermediate subproblems.The effective and lifted gradient information matrices formalize the comparison.
Metric recovery: The smallest eigenvalue of the lifted effective gradient information matrix measures the weakest useful gradient signal.This provides the geometric quantity used to assess non-degenerate learning information.
Gradient dead zone: Outcome-based GRPO can enter a gradient dead zone when the probability of a correct rollout is below δ, because reward groups collapse and the worst-case effective gradient vanishes.The analysis defines p(x; θ) as the current policy’s probability of solving x.
Metric recovery: Theorem 4.3 states that subproblem curricula recover a non-degenerate learning geometry even when the original problem provides almost no useful gradient signal.The result assumes intermediate subproblem success probabilities lie between p⋆ and 1 − p⋆ for j < K, together with conditional identifiability.
Difficulty dependence: The recovery ratio grows as p(x; θ) approaches zero, predicting larger relative gains on harder problems.The empirical results are reported as consistent with this theoretical prediction.

5. Experiment

Experiments across seven mathematical reasoning benchmarks show that SCRL consistently improves performance, hard-problem solving, and credit assignment over strong baselines. Further analyses indicate gains remain with weaker subproblem generators and transfer to direct problem solving.

Main Results: SCRL reaches 35.0% average accuracy on Qwen3-4B, exceeding QuestA by 3.0 points and vanilla GRPO by 4.1 points.It achieves the best average performance across all tested model scales and benchmarks.
Hard-Problem Solving: SCRL consistently outperforms GRPO and other curriculum RL baselines across the evaluated pass@k range on AIME24, AIME25, and IMO-Bench.The result indicates stronger hard-problem solving across different sampling budgets.
Hard-Problem Solving: SCRL achieves a higher solvable-problem ratio than GRPO under both full-group and half-group protocols.This shows curriculum progress transfers to direct hard-problem solving rather than remaining confined to curriculum-format rollouts.
Generator Quality: SCRL improves over GRPO by +2.7 points with Qwen3-4B-Instruct subproblems and by +3.9 points with DeepSeek-V3.2 subproblems.The method remains effective with a weaker generator, while better subproblem quality can further increase gains.
Credit Assignment: Subproblem-level normalization with progress-aware correction performs best in the credit-assignment ablation.Without correction, later steps may be rewarded after earlier failures; Both-GRPO cannot credit valid intermediate progress.

6. Conclusion

SCRL converts hard reasoning problems into verifiable subproblem curricula and uses subproblem-level normalization for fine-grained credit assignment. Theory and experiments show that this decomposition lifts problems out of gradient dead zones and improves performance over strong baselines.

Conclusion: SCRL derives verifiable subproblems from reasoning chains and converts partial rollout progress into token-level learning signals.This enables fine-grained credit assignment without external reward models or process annotations.
Conclusion: Subproblem decomposition lifts hard problems out of gradient dead zones, while experiments show consistent gains over strong RLVR and curriculum-learning baselines.The recovery effect is predicted theoretically and supported empirically.

A. More Ablation Study

The ablations examine how curriculum length and training-data construction affect SCRL. They show that K = 4 balances denser supervision with rollout complexity, while integrated subproblem curricula outperform simply adding or isolating questions.

Number of subproblems: K = 4 is used as a practical trade-off between denser supervision and curriculum complexity.Longer curricula increase rollout complexity and can block later credit when intermediate subproblems are ambiguous or poorly constructed.
Training data construction: SCRL outperforms both data-scaling controls in the training-data construction ablation.The controls expand original-problem data or split curriculum instances into standalone questions.
Training data construction: SCRL’s four-subproblem rollouts have an average response length of only about 1.5× GRPO on hard_1024.This indicates that response length does not grow proportionally with the number of subproblems.
Training data construction: Simply adding more hard problems is less effective than preserving dependent subproblems within each rollout under the same training budget.hard_4096 improves over hard_1024, while subproblem_4096 still falls behind SCRL because isolated subproblems do not teach the harder target-solving process as effectively.

B.1. Proof of Theorem 4.2 (Bound on λmin(Fx(θ)))

The proof explains why direct RLVR can lose its effective gradient signal on hard problems. When reward groups are degenerate, GRPO advantages vanish; non-degenerate groups occur with probability bounded by the rarity of successful rollouts.

Fisher-information bound: The proof bounds λmin(Fx(θ)) by separating degenerate and non-degenerate reward-group events.It conditions gradient terms on reward vectors and analyzes the contribution from a non-degenerate first column.
Degenerate reward groups: On the event that all G rollouts share the same reward, the GRPO group is degenerate and every advantage is zero.This makes every term in the corresponding gradient contribution vanish.
Probability bound: For binary rewards, the probability of a non-degenerate reward group is at most Gδ when the success probability p(x; θ) is below δ.The bound follows from 1 − (1 − p)^G ≤ 1 − (1 − δ)^G ≤ Gδ.
Advantage control: For a non-degenerate group with k successes, the normalized advantage depends on the success fraction k/G and its standard deviation.The proof separately evaluates success and failure rollout advantages using binary rewards.

C. SCRL Training Algorithm

SCRL mixes curriculum and original-problem rollouts within each training step. It verifies subproblem answers, computes progress-aware rewards and position-specific advantages, then updates the policy using both curriculum and standard GRPO signals.

Curriculum rollouts: Each training step samples a problem and forms its K-subproblem curriculum prompt.The curriculum is drawn from a problem set, policy, subproblem bank, subproblem count K, and group size G.
Curriculum rollouts: SCRL samples G/2 curriculum rollouts and extracts K tagged answer spans from each response.Each subproblem answer is verified to produce a raw reward vector.
Reward computation: Progress-aware correction transforms subproblem rewards before final-subproblem rewards are assigned across all K positions.The algorithm computes curriculum progress and then sets the final subproblem rewards for each rollout.
Credit assignment: Subproblem-level advantages are computed separately for each position and assigned to the corresponding tagged answer-span tokens.This gives curriculum rollout tokens position-specific advantages.
Mixed training: The remaining G/2 original-problem rollouts use verified final answers and standard GRPO advantages before the policy update.SCRL therefore combines curriculum and original-problem training signals in the same objective.

D. OOD Task Performance

SCRL transfers beyond the mathematical benchmarks used to construct its curricula. On GPQA, HumanEval, and LiveCodeBench v6, it achieves the best average OOD score and improves over both the base model and GRPO on every benchmark.

Overall OOD performance: 51.67 is SCRL’s average OOD score, above 47.20 for the base model and 48.37 for GRPO.The evaluation uses Qwen3-14B-Base on GPQA, HumanEval, and LiveCodeBench v6.
Interpretation: The OOD results suggest that SCRL improves transferable reasoning behavior rather than merely overfitting to curriculum prompts or the training distribution.The evaluated domains include multi-step reasoning and program synthesis.

E. Detailed Experimental Results

The experiments report complete pass@k results for Qwen3-4B-Base and use a consistent evaluation protocol across methods, with configurations documented separately.

Complete pass@k performance is reported for k ∈ {1, 2, 4, 8, 16, 32, 64} on Qwen3-4B-Base.
Hyperparameter configurations and training settings are documented in dedicated tables, with experiments trained using the Verl framework.
Evaluation selects each baseline’s checkpoint using its best average validation score before testing.
Each test problem receives n = 64 generated rollouts, and pass@k is computed with the stated estimator.

F.3. Baseline Implementation Details

The implementation compares SCRL with supervised and reinforcement-learning baselines, uses an LLM-generated progressive curriculum, and evaluates structured multi-subproblem prompting alongside the original task.

Baseline Implementation Details: SFT fine-tunes models on detailed reasoning trajectories synthesized through the DeepSeek V3.2 API.
Baseline Implementation Details: GRPO uses the standard Group Relative Policy Optimization implementation without additional reward shaping or gradient modification terms.
Baseline Implementation Details: QuestA and NuRL provide curriculum-based RL baselines using progressively reduced hints or self-generated abstract cues.QuestA uses partial-50 and partial-25 phases, while NuRL separates GRPO and cue-based training stages.
Limitations and Future Work: SCRL’s limitations are dependence on an external LLM for subproblem construction and reliance on verifiable answers, restricting direct application to open-ended tasks.
Subproblem Curriculum: The curriculum generator is instructed to produce exactly four self-contained subproblems with strictly increasing difficulty and clean numerical ground truths.
Chat Templates: Curriculum prompts require solving four problems in order, with exactly four tagged reasoning blocks and a boxed final answer for each.