PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Decomposed Subtasks

Mattie Terzolo, Mikolaj Sacha, Ayan Sinha, Andrew Rabinovich

arXiv:2609.27035v2cs.AIcs.LG

TL;DR

Scalar-reward policy updates collapse multi-turn outcomes, leaving competency-specific failures and the trajectory segments responsible for them indistinguishable to the optimizer. RLDS uses a fixed task taxonomy to allocate trajectory reward across subtasks, compute group-relative subtask advantages, and concentrate token credit around reflection-identified pivots. Gains scale with subtask heterogeneity, with the largest gain on ScienceWorld (+11.5 points over RLVR); HotpotQA and DeepResearch regress toward scalar RLVR.

  • Problem

    Scalar-reward policy updates collapse multi-turn outcomes, leaving competency-specific failures and the trajectory segments responsible for them indistinguishable to the optimizer.

    Source: §1 · pp. 1–2

  • Method

    RLDS uses a fixed task taxonomy to allocate trajectory reward across subtasks, compute group-relative subtask advantages, and concentrate token credit around reflection-identified pivots.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Gains scale with subtask heterogeneity, with the largest gain on ScienceWorld (+11.5 points over RLVR); HotpotQA and DeepResearch regress toward scalar RLVR.

    Source: §5 · p. 6

  • Takeaways & Limitations

    Training-time diagnostics identify where subtask decomposition recovers signal, and ScienceWorld runs at 0.89× scalar GRPO’s wall-clock time per step.

    Source: §5.2 · p. 8, §7 · p. 10

    Task taxonomies require practitioner iteration, and static subtask importance weights must be tuned by the practitioner.

    Source: §6 · p. 9

Abstract

from arXiv · show

Group Relative Policy Optimization (GRPO) and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills, especially under sparse and delayed environmental feedback, this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how that should change behavior. We argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks (RLDS), whose core is Subtask-Decomposed Advantage Estimation (SDAE): a replacement for the scalar GRPO advantage that splits trajectory reward into per-subtask shares on a fixed taxonomy, computes a group-relative advantage per subtask, and distributes per-token credit by weighting each subtask's advantage by its importance, concentrating it around the step where a reflection marks that subtask's execution as consequential. We evaluate on four agentic benchmarks: FrozenLake (sparse grid navigation), HotpotQA (multi-hop QA, one retrieval tool), ScienceWorld (long-horizon embodied science), and DeepResearch (long-form research, four tools, composite rubric reward). Heterogeneity diagnostics emitted during training show where decomposition pays off - gains scale with subtask heterogeneity, largest on the high-heterogeneity tasks ScienceWorld (+11.5 points, paired-bootstrap 95% CI [+9.8, +13.3]) and FrozenLake (+9.8 points, [+7.0, +12.8]), and within noise on HotpotQA and DeepResearch, where the diagnostics predicted little to recover. ScienceWorld is also more compute-efficient under RLDS than scalar GRPO (-10.9% wall-clock per step), as long rollouts amortize the fixed reflect-and-grade overhead.

1 Introduction

Scalar GRPO gives identical credit to rollouts with equal reward even when their competency profiles and failure points differ. RLDS preserves subtask-level judgments and pivot locations in the policy update, and evaluates whether gains track task heterogeneity.

1 Introduction

  • Two ScienceWorld rollouts with reward 0.4 can fail for different reasons, yet scalar GRPO assigns both the same advantage and broadcasts it across every token.That broadcast can punish correctly gathered items when the separate failure was interpreting the results.Source: §1 · p. 1
  • A single trajectory reward cannot distinguish which competency needs improvement or which segment earned credit, even when a grader can identify the failure.The paper frames this as an informational bottleneck on multi-dimensional tasks, following a position paper.Source: §1 · p. 2
  • RLDS replaces scalar credit with group-relative advantages for each subtask and concentrates each subtask’s token credit around its reflection-identified pivot.The resulting per-token gradient varies across both competencies and positions, unlike GRPO’s uniform broadcast.Source: §1 · p. 2
  • The evaluation tests whether gains track within-group subtask-signal variation and the spread of competency pivots across a trajectory.These are the two forms of variation that scalar GRPO’s trajectory-level reward and uniform token broadcast average away.Source: §1 · p. 2

2 Related Work

RLDS combines reflection-based training with within-trajectory credit assignment. Prior approaches either compress reflection into scalar reward or redistribute scalar credit without an explicit competency decomposition.

2 Related Work

  • RLDS is positioned at the intersection of reflection-based RL and token- or step-level credit assignment.The related work reviews both lines as foundations for the method.Source: §2 · p. 2

2.1 Reflection-Based RL

  • ERL adds reflect-then-retry rollouts and response-side distillation, while R3L uses structured reports, retry pivots, and additional gradient-shaping mechanisms.RLDS adopts ERL’s retry-distillation channel and R3L’s report, pivot mask, and reflection-NLL term, while dropping R3L’s OPMD baseline and positive amplification.Source: §2.1 · pp. 2–3

2.2 Token- and Step-Level Credit Assignment

  • GTPO, GRPO-S, and GRPO-λ redistribute scalar reward using entropy or eligibility traces, while process reward models score steps and AgentPRM extends step supervision to agentic rollouts.These approaches do not decompose credit by competency; process reward models also require annotation or a trained verifier.Source: §2.2 · p. 3

3 Reinforcement Learning with Decomposed Subtasks

RLDS uses reflection-generated competency judgments and decisive-turn pivots to decompose reward before the policy update. SDAE computes group-relative subtask advantages, recombines them with importance weights, and distributes token credit around each pivot; the combined objective also retains auxiliary distillation.

3.1 Motivation

  • RLDS consumes reflection-generated diagnoses by decomposing reward across competencies and routing credit to the turn identified as decisive.Its SDAE estimator and trajectory-wide reward-distribution mechanism address competency and temporal credit assignment together.Source: §3.1 · p. 3

3.2 Preliminaries: Subtask Self-Grading on Reflect-Retry Rollouts

  • Each task uses a fixed subtask taxonomy with static importance weights; the policy self-grades rollouts and retries with per-subtask grades and pivot steps.The same policy acts as actor, grader, and reflector, while the grades allocate—but do not scale—the environment reward across subtasks.Source: §3.2 · pp. 3–4

3.3 Subtask-Decomposed Advantage Estimation

  • SDAE allocates trajectory reward to subtasks using zero-anchored grade proportions, then computes a group-relative advantage for each subtask.Separate subtask baselines expose whether a rollout is above or below its group on each competency, even when its total reward is at the group mean.Source: §3.3 · p. 4
  • Static importance weights recombine subtask advantages into a row-level scalar, which is batch-normalized; missing grades fall back to vanilla GRPO advantages.Training logs sign disagreement, correlation with GRPO, and token-level advantage spread as diagnostics of recovered multidimensional signal.Source: §3.3 · pp. 4–5
  • When pivots are available, SDAE applies each subtask advantage through a kernel centered on its decisive turn rather than uniformly broadcasting credit across tokens.The kernel redistributes credit without changing the per-row total and composes with R3L’s mask to focus retry gradients on the diverging suffix.Source: §3.3 · p. 5

3.4 Combined Objective

  • The combined training objective pairs a PPO surrogate using SDAE per-token advantages with an auxiliary distillation loss.The auxiliary loss includes reflection-NLL and retry-response distillation terms, weighted internally by retry reward delta.Source: §3.2 · p. 4, §3.4 · p. 5

4 Experiments

The experiments compare RLDS with scalar GRPO and a reflect-retry baseline across four agentic benchmarks spanning different task structures. Reflection-enabled conditions use grouped originals and retries, with task-specific setups and rewards.

4 Experiments

  • RLDS is compared with scalar GRPO and reflect-retry training using vanilla GRPO advantages across four benchmarks selected to span subtask heterogeneity.Reflection-enabled conditions use six originals and up to three retries, folded into a group of size G+R to match the per-step token budget.Source: §4 · p. 6
  • The benchmarks range from 10-turn sparse FrozenLake navigation to 30-turn ScienceWorld procedures, alongside five-turn HotpotQA and 10-turn DeepResearch.They vary in tools and rewards: HotpotQA uses one search tool and F1, ScienceWorld uses simulator partial credit, and DeepResearch uses four tools and a rubric plus formatting scores.Source: §4 · p. 6

5 Analysis

RLDS gains most when tasks distribute competencies across longer horizons, where subtask signals vary and pivotal steps span the rollout. Its diagnostics track whether decomposition changes rollout weighting and token-level credit, while task-level readouts show which competencies improve and when they matter.

5 Analysis

  • +11.5 points over RLVR is RLDS’s largest reported gain, on ScienceWorld, while gains scale with process-heavy, long-horizon tasks.HotpotQA and DeepResearch have minimal process content over short horizons, so their SDAE advantages regress toward scalar RLVR.Source: Figure 2 · p. 6, §5 · p. 6

5.1 SDAE Metrics

  • Sign disagreement, GRPO correlation, and token-level advantage standard deviation track whether subtask signals vary, decomposition reweights rollouts, and pivot masks redistribute token credit.ScienceWorld clears all three gates; HotpotQA and DeepResearch wash out at the first, with SDAE regressing toward scalar RLVR.Source: §5.1 · p. 7

5.2 Compute Profile

  • At long horizons, ScienceWorld’s RLDS step time is 0.89× RLVR despite more work per trajectory; reward gains are measured at fixed step count.Short-horizon step time rises 2.4×/2.3×/1.6× on FrozenLake/HotpotQA/DeepResearch; a budget-matched comparison remains future work.Source: §5.2 · p. 8, Table 1 · p. 8

5.3 Subtask Metrics

  • FrozenLake retries most improve hazard_identification (near +1.8), while dead_end_detection gains sharply late; its pivot lies deeper than early hazard_identification and path_planning pivots.The late improvement occurs as remaining failures involve tricky maps where the agent could get stuck.Source: §5.3 · p. 8, Figure 6 · p. 8
  • ScienceWorld’s subtask importance shifts toward scientific_reasoning around step 200, after the model learns to interact with the environment.Pivots span the 30-turn rollout: goal_decomposition ≈2, exploration_efficiency and object_identification ≈5, and action_sequencing, scientific_reasoning, and state_tracking ≈11–13.Source: §5.3 · p. 9, Figure 7 · p. 9

6 Limitations

RLDS has two practitioner-facing constraints: subtask taxonomies should follow trajectory stages, and subtask importance weights remain fixed and require manual tuning.

Defining subtasks

  • Taxonomies should align subtasks with temporal stages; orthogonal competencies can collapse pivot steps onto the same turn, making SDAE resemble scalar broadcast.HotpotQA’s search_planning → search_iteration → evidence_extraction → answer_grounding is an example of stage alignment.Source: §6 · p. 9

Static subtask weights

  • Importance weights φk are static per task and must be tuned by the practitioner; dynamic weighting from retry-grade deltas is proposed as an extension.The proposed extension uses a Dirichlet prior, updating weights from observed deltas so consistently helpful subtasks gain weight while saturated ones decay.Source: §6 · p. 9

7 Conclusion

RLDS uses competency-tagged diagnoses and decisive-turn pivots to distribute credit by subtask and position. Its gains track measured heterogeneity: they are largest on ScienceWorld and FrozenLake, while HotpotQA and DeepResearch remain within noise.

Conclusion

  • +26.7% lift on ScienceWorld and +11.7% on FrozenLake; HotpotQA and DeepResearch lifts sit in noise.ScienceWorld’s RLDS step costs 0.89× a scalar GRPO step because retries resample only the suffix from the pivot.Source: §7 · p. 9
  • RLDS preserves directional disagreements through group-relative per-subtask baselines and reshapes token credit around grader-marked decisive turns.The diagnosis comes from reflect-retry rollouts, which identify what failed and which turn determined each failure.Source: §7 · p. 9
  • Gains track within-group variation in per-subtask signals and the spread of competency pivots across the trajectory.ScienceWorld shows sign disagreement throughout training and pivots spanning approximately turns 2–13 of a 30-turn budget.Source: §7 · p. 9
  • Sign-disagreement rate, GRPO correlation, and token-level advantage standard deviation locate tasks along the heterogeneity gradient; retry deltas and pivot steps provide a continuous training readout.The paper concludes that when rollouts already produce competency-tagged signals, the advantage estimator should consume them.Source: §7 · p. 10

A 5.3 Subtask Metrics continued

The per-subtask metrics show different temporal patterns across the two continued benchmark analyses: DeepResearch separates skill pivots across its trajectory, while HotpotQA’s retry-grade gains fade quickly.

DeepResearch

  • DeepResearch pivots progress from retrieval_strategy at ≈1.5, through analytical_reasoning and factual_accuracy at ≈2.5–3, to completeness_coverage and citation_practice at ≈3.5–4.Retry-grade deltas climb sharply after step 200, led by factual_accuracy (peaks ≈2.0), while citation_practice stays near zero; Figure 8 plots retry grade deltas and pivot step.Source: A 5.3 Subtask Metrics continued · p. 11, Figure 8 · p. 11

HotpotQA

  • HotpotQA’s retry-grade deltas fade quickly and settle into a small positive gain; Figure 9 visualizes its four-skill taxonomy and per-subtask metrics.The passage relates this pattern to initial validation-reward over-performance that fades by step 50.Source: A 5.3 Subtask Metrics continued · p. 11, Figure 9 · p. 11

B Training Runs

Training uses a shared GRPO recipe with task-specific run settings and reports per-task run statistics. RLDS adds reflection and a per-subtask pivot path to the reflection-augmented conditions.

Shared training recipe

  • All runs fine-tune Qwen3-4B-Instruct-2507 with GRPO via verl and SGLang, using AdamW at 1×10−6 and one PPO epoch per step.Runs use one node with 8×H100 80 GB and report the best validation reward across training.Source: B Training Runs · p. 11

Per-task hyperparameters and run statistics

  • Table 2 reports task-level dataset sizes, differing runtime settings, update steps, step and run times, hardware, and external-API call volumes.Slash-separated cells compare RLVR / RLDS where conditions differ; dollar costs are omitted because they depend on contract pricing.Source: Table 2 · p. 12, B Training Runs · p. 12
  • ScienceWorld trains for 20 epochs on 1,796 examples; FrozenLake and HotpotQA each make one pass over 20k examples, while DeepResearch runs two epochs on 6,480 examples.ScienceWorld’s fresh JVM seed per example adds parametric variation; DeepResearch stops at a 315-step wall-clock cap.Source: B Training Runs · p. 12

R3L / RLDS knobs

  • Reflection is triggered on the bottom 50% of failures per group, and RLDS enables per-subtask pivots with a Gaussian kernel (σ=1.0 turn, ℓ1-normalized per row).The policy acts as grader and reflector; auxiliary SFT and retry-distillation coefficients are 0.05 and 0.10, respectively.Source: B Training Runs · p. 12

C Subtask Decompositions

The section specifies a fixed per-task subtask taxonomy that supplies the names and importance weights used by the grader and SDAE.

C Subtask Decompositions

  • Each task’s taxonomy lists grader-facing subtask names and descriptions plus static SDAE importance weights that sum to 1.0.Subtask names match the grader’s JSON output and the logged per-subtask metrics.Source: C Subtask Decompositions · pp. 12–13

C.1 FrozenLake

The four task taxonomies divide competencies along distinct skill and temporal dimensions, with subtask counts and rollout budgets varying by benchmark.

C.1 FrozenLake

  • FrozenLake uses five subtasks over 10 turns to separate spatial planning and boundary awareness from hazard reasoning and execution-state tracking.The taxonomy lets the grader identify the decisive turn separately for each competency.Source: §C.1 · p. 13, Table 3 · p. 13

C.2 HotpotQA

  • HotpotQA uses four subtasks over five turns for the plan–retrieve–extract–ground pipeline, with planning pivots early and grounding pivots late.The temporal ordering is shown in Figure 9.Source: §C.2 · p. 13, Table 4 · p. 13

C.3 ScienceWorld

  • ScienceWorld uses six subtasks over 30 turns, spanning early planning, mid-trajectory enactment, and late verification.It has the suite’s longest horizon and widest spread of pivot-step means.Source: §C.3 · p. 13, Table 5 · p. 13

C.4 DeepResearch

  • DeepResearch uses five subtasks over 10 turns, separating search and source handling while treating synthesis as a cross-cutting completeness signal.The temporal stages emerge in Figure 8.Source: §C.4 · p. 14, Table 6 · p. 14

D Benchmark Details

The benchmark details specify each environment’s interaction and reward setup, while the reflection prompts collect competency grades and outcome-aware pivotal steps to guide retries.

D.1 FrozenLake

  • FrozenLake uses free-form cardinal-move actions on abstract-symbol grids and provides only a sparse binary reward: +1 for reaching the goal and 0 otherwise.Maps are procedurally generated, and episodes end on reaching the goal, entering a hole, exhausting the 10-turn budget, or insufficient response-token budget.Source: §D.1 · p. 14

D.3 ScienceWorld

  • ScienceWorld provides natural-language observations and a 0–1 normalized monotonic-max partial-credit score, modified by discounts and shaping terms.A no-experiment discount applies to qualifying procedure-required tasks, while classification task families are exempt; episodes have a 30-turn budget.Source: §D.3 · pp. 15–16

D.4 DeepResearch

  • DeepResearch combines a weighted rubric outcome with citation, format, and search signals in its reward: r = 0.7 rrubric + 0.1 rcitation + 0.1 rformat + 0.1 rsearch.The dataset uses static per-instance rubrics and intentionally omits DR Tulu’s evolving-rubric mechanism [9].Source: §D.4 · p. 17

E Reflection and Grading Prompts

  • The grader assigns each competency a 1–5 score and an outcome-aware pivot_step: the earliest load-bearing step on success or earliest degradation step on failure.The reflection prompt then uses the grades to identify subtasks to address and produce an actionable retry suggestion.Source: E Reflection and Grading Prompts · p. 18
  • In the FrozenLake example, an empty Game Rules block leaves the policy without the C-is-a-hole prior, and grading identifies hazard_identification as the failed competency before the reflector supplies a rule for retry.The figure describes the hazard competency as having the highest weight after path_planning.Source: Figure 10 · p. 20
Loading 2609.27035v2…