PaperlayerView exact revision on arXiv

Source-linked AI summary

Dependency-Aware Reward Shaping for Agentic Reinforcement Learning

Ziyi Chen, Yan Zhang, Jianhui Wei, Daoan Zhang, Zuozhu Liu

arXiv:2610.01207v1cs.AI

TL;DR

Outcome-only rewards make useful progress hard to distinguish from mistakes, and existing step-credit approaches do not directly represent prerequisite dependencies. DARS assigns signed step rewards from annotated predicate changes over a dependency graph. Across five task families, it improves or matches results across agentic and reasoning settings, with ablations supporting the contribution of its credit components.

  • Problem

    Outcome rewards provide little step-level guidance, and failed episodes leave useful progress indistinguishable from mistakes; temporal or uncertainty-based credit does not directly encode prerequisite dependencies.

    Source: §1 · pp. 1–2

  • Method

    DARS uses an annotator's verification, invalidation, and repair events to assign signed step rewards from a fixed potential over prerequisite-linked task predicates.

    Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 6

  • Results

    Across five task families, DARS improves ALFWorld and Search-R1 results, raises WebShop scores, works with GiGPO and AEPO, and achieves the highest average among compared tool-free reasoning methods.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Ablations show that step-level credit, dependency attenuation, and graph topology each contribute to performance.

    Source: §5.4 · p. 8, §6 · p. 9

    The dependency-bounded invalidation claim assumes the task's predicate-state abstraction, and repairing a prerequisite restores only descendants whose local evidence remains valid.

    Source: §1 · p. 2, §4.2 · p. 5

Abstract

from arXiv · show

When training large language models with reinforcement learning, terminal rewards provide little guidance about which steps matter. Common methods for assigning step credit overlook that work built on uncorrected mistakes is wasted while independent work remains valid. With only a final success/failure reward, every step in a failed episode has zero total future reward, even when it made progress. We propose Dependency-Aware Reward Shaping (DARS), which represents task progress as predicates linked by prerequisite relations and assigns step-level credit over the dependency graph. An annotator marks which predicates each step verifies, invalidates, or repairs. Verified predicates are discounted according to graph distance from the nearest broken prerequisite, while independent predicates are unaffected. Repairs update these weights based on any errors that remain; invalidated predicates need re-verification to regain credit. A fixed potential converts these annotations into signed per-step rewards. A common reward and annotation interface allows DARS to integrate with a range of reasoning and agentic training methods, such as GiGPO and ARPO/AEPO, without changing their rollout strategies or optimizers. Across five task families and models from 1.5B to 8B, DARS improves success by up to 10 points over GiGPO trained with the same budget and harness (ALFWorld), raises the WebShop task score and Search-R1 QA accuracy, complements AEPO's entropy-based training on AIME24/25 with a Python interpreter, and exceeds OmniOPD in controlled tool-free reasoning comparisons at 1.7B and 4B. Ablations show that step-level credit, dependency attenuation, and graph topology each contribute. On ALFWorld, a distilled 8B annotator matches the API annotator, enabling DARS to run efficiently without a frontier judge. Code is available at https://github.com/JianhuiWei7/DARS.

1 INTRODUCTION

DARS addresses sparse outcome credit by tracking task predicates and their prerequisite relations, shaping signed step rewards according to valid progress, dependency, and repair. It is evaluated across five task families and integrates with existing learners without changing their rollout strategies or optimizers.

1 INTRODUCTION

  • Outcome-only rewards give useful intermediate decisions and irrelevant detours the same credit, while failed trajectories leave useful progress indistinguishable from mistakes.Recent uncertainty- and time-based credit methods do not directly encode which subgoals a step establishes or which later decisions depend on it.Source: §1 · pp. 1–2
  • Across five task families, DARS improves ALFWorld and WebShop results, raises Search-R1 accuracy, complements AEPO on AIME24/25, and exceeds OmniOPD in tool-free reasoning comparisons.The paper reports a 4.2-point increase in AEPO's best evaluated AIME24/25 score.Source: §1 · pp. 2–3
  • Under the task's predicate-state abstraction, an unrepaired error invalidates work according to dependency structure, leaving only the dependent suffix off-distribution for the target policy.DARS attenuates credit for dependent predicates, preserves independent branches, and restores credit after a prerequisite is repaired and re-verified.Source: §1 · p. 2
  • DARS assigns signed step rewards from a fixed dependency-graph potential over annotated task predicates, and can modify existing RL reward signals without changing rollouts or optimizers.An annotator marks which predicates each step verifies, invalidates, or repairs; the potential weights supported predicates by distance from the nearest broken ancestor.Source: §1 · pp. 2–3
  • The paper presents DARS as a signed, auditable reward from a fixed graph potential, with a decay parameter spanning milestone counting to hard masking below a break.It reports integrating the interface with GiGPO, ARPO/AEPO, and a token-level interface without changing their rollouts or optimizers.Source: §1 · p. 3

2 RELATED WORK

The related work spans group-relative RL, intermediate-step evaluation, and credit assignment beyond temporal position. DARS is distinguished by deriving credit from a graph of task predicates and their dependencies.

Reinforcement learning for LLM agents

  • Group-relative RL compares rollouts without a learned value function; GiGPO (Feng et al., 2025) adds recurring-state comparisons, while ARPO (Dong et al., 2026b) and AEPO (Dong et al., 2026a) allocate samples using token entropy after tool calls.DARS supplies a reward signal that composes with these mechanisms and is evaluated atop GiGPO and AEPO.Source: §2 · p. 3

Process rewards and LLM judges

  • Process reward models and LLM evaluators assess intermediate steps or attribute failures, whereas DARS asks an annotator for structural events and computes rewards deterministically from them.The same annotations can be re-credited offline under different rules.Source: §2 · p. 3

Credit beyond temporal position

  • Prior credit methods include return decomposition, counterfactual analysis, and agentic stepwise or graph-based attribution; DARS instead computes credit from dependency-weighted progress over a task-predicate graph.Its representation distinguishes dependent work, independent branches, and post-error recovery.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe sparse outcome rewards and group-based advantages, then distinguish an episode-level signal from a step channel that DARS can densify. GiGPO adds comparisons at recurring states, while the episode channel and optimizer can remain unchanged.

Problem setup

  • Under strict outcome rewards, only successful terminal segments receive reward, so failed episodes have essentially all-zero reward sequences that do not distinguish useful from incorrect actions.A segment may be an environment turn or a span produced by a fixed segmentation operator; segments that change nothing receive no immediate shaping reward.Source: §3 · pp. 3–4

Group-based RL

  • Group-based RL normalizes trajectory returns within a task group to form an episode-level advantage shared by every segment.GRPO-style methods compare sampled trajectories within the group rather than against a learned value function.Source: §3 · p. 4
  • GiGPO adds a finer signal by comparing actions from the same recurring state using discounted future returns.If no state recurs, all anchor groups are singletons, the state-level advantage is zero, and the combined advantage reduces to the episode-level advantage.Source: §3 · p. 4

Two channels

  • DARS writes dense rewards into the step channel while retaining the episode channel, anchor grouping, and optimizer; learners without anchor comparisons can use a token-level interface.The two channels are the per-segment reward sequence and the episode return, respectively.Source: §3 · p. 4

4 DARS

DARS assigns step-level credit from changes in dependency-weighted task progress, using annotated predicates and prerequisite relations. Persistent graph state and a fixed potential turn those annotations into signed rewards for compatible learning interfaces.

4 DARS

  • DARS assigns time-step credit from changes in dependency-weighted progress while tracking the persistent state of each task predicate.The task is represented as predicates connected by prerequisite relations.Source: §4 · p. 4, §4.1 · p. 4

4.1 SOLUTION TRACES AND DEPENDENCY GRAPHS

  • Task requirements form a dependency graph, and an annotator marks affected predicates with verification, invalidation, or repair events.Repair alone does not re-establish a predicate; the step must also verify it, or verification must occur later.Source: §4.1 · pp. 4–5

4.2 FROM GRAPH STATE TO STEP CREDIT

  • The persistent state distinguishes verified predicates from broken or unestablished ones, and preserves each state until an annotation changes it.Breaking an ancestor attenuates a descendant without deleting its local evidence; an independently incorrect descendant must be marked broken separately.Source: §4.2 · p. 5
  • Verified predicates contribute to a fixed potential, with credit attenuated by the nearest unresolved broken ancestor while unrelated branches remain unaffected.Repair lifts a break, but predicates regain support only through verification; descendants retain full weight only when their local evidence remains valid.Source: §4.2 · pp. 5–6

4.3 ANNOTATION AND LEARNING INTERFACE

  • A learner can use DARS rewards without changing its rollout strategy or optimization loss, including through a direct token-level interface.The interface can combine local credit with group-relative trajectory credit; setting η = 0 gives an annotation-matched trajectory-level control on fixed traces.Source: §4.3 · p. 6

5 EXPERIMENTS

The experiments evaluate DARS across interactive agents, agentic mathematics, and tool-free reasoning, then test its components through ablations.

5.1 SETUP

  • The evaluation covers five task families, using GiGPO for interactive tasks, ARPO/AEPO for agentic mathematics, and a token interface for tool-free reasoning.Baselines include published references and local reproductions, outcome-only ARPO/AEPO arms, and tool-free distillation and process-feedback methods.Source: §5.1 · pp. 6–7

5.2 INTERACTIVE AGENT RESULTS

  • ALFWorld 1.5B DARS finishes at 96.9% success versus 86.9% for GiGPO, narrowing the gap to 7B policies.GiGPO peaks at step 200, while DARS improves again after step 300; these trajectories come from single training runs.Source: §5.2 · p. 7, Table 1 · p. 7
  • WebShop 7B DARS retains a 1.5-point success advantage over equally extended GiGPO, while held-out Search-R1 reaches 41.6% versus 39.5% in all decoding groups.Search-R1 results support the need to establish answer prerequisites and convert that progress into an answer; larger multi-hop gains vary across datasets, and 2Wiki declines.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 MATHEMATICAL REASONING

  • AEPO + DARS raises the grid mean from 61.7% to 63.6%, while ARPO + DARS falls from 62.0% to 61.6%.The combined AEPO arm does not improve every checkpoint, and the authors note approximately ±5 points of decoding variation per cell.Source: §5.3 · p. 8, Table 2 · p. 8
  • DARS has the highest mean at both tool-free reasoning scales and exceeds OmniOPD at 1.7B (p = 0.031) and 4B (p = 0.005 on the primary host).The same-annotation scalar control indicates that step-level credit matters clearly at 1.7B, but its effect is unresolved at 4B.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 ABLATIONS

  • Scalarizing credit, removing attenuation, and replacing WebShop’s requirement graph with a chain each reduce performance.The ablations test whether step placement, discounting dependent progress, and representing task topology contribute to DARS’s results.Source: §5.4 · p. 8, Table 4 · p. 9
  • A distilled Qwen3-8B annotator reaches 92.2% peak success versus 89.1% for its teacher, a difference within decoding noise.It reproduces the teacher’s step-level credit at r = 0.81; the comparison supports practical sufficiency, not superiority.Source: §5.4 · p. 9, Table 4 · p. 9

6 CONCLUSION AND LIMITATIONS

DARS assigns credit over task predicates linked by prerequisite relations, with verification, invalidation, and repair events replayed into signed step rewards. The paper reports benefits across its evaluated tasks but leaves scaling and open-ended tasks with unclear dependencies unresolved.

6 CONCLUSION AND LIMITATIONS

  • DARS uses dependency graphs to assign signed step rewards, and its evaluation identifies scaling and open-ended tasks with unclear dependencies as unresolved boundaries.The conclusion reports ablations supporting step-level credit, dependency attenuation, and graph topology.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

  • The reproducibility statement describes training settings, annotation prompts, replay procedures, audits, costs, per-benchmark results, and recorded trajectories in appendices.It says code for the annotator, reward kernel, learner integrations, and persisted graphs will be released.Source: REPRODUCIBILITY STATEMENT · p. 10

A ANNOTATION CONTRACT, PROMPTS, AND STATE REPLAY

  • An annotator returns per-step verified, error, and recovered predicate events; a shared deterministic replay procedure computes the potential from those annotations.The event structure is supplied without a numeric reward from the annotator; replay is shared across domains.Source: A ANNOTATION CONTRACT, PROMPTS, AND… · pp. 13–14
  • Predicate weights use distance to the nearest broken ancestor, while verification, invalidation, and repair events update support and broken state.The deployed rule is reported to produce the same potentials as strongest attenuation on all WebShop graphs and most ALFWorld and mathematics graphs.Source: A ANNOTATION CONTRACT, PROMPTS, AND… · p. 14
  • Interactive tasks use environment turns, tool-free responses use eight paragraph-aligned blocks, and Python-interpreter mathematics follows ARPO turns.Domain topologies include independent instance chains in ALFWorld, requirement graphs in WebShop, and hop or derivation chains in Search and mathematics.Source: A ANNOTATION CONTRACT, PROMPTS, AND… · pp. 14–15

B ANNOTATION QUALITY

The annotation-quality section audits event accuracy, repeatability, reward discrimination, rendering alignment, and a distilled annotator. The distilled model matches the API annotator’s ALFWorld performance in the reported comparison.

Manual audit

  • In a 54-annotation manual audit, reviewers judged 34 correct, 12 minor-error, and 8 major-error annotations.Across roughly 15 identity traps, no reviewer found positive credit for a wrong object.Source: B ANNOTATION QUALITY · p. 17

Test–retest reliability

  • Verification events had the highest test–retest agreement (κ = 0.84), while repairs were the rarest and least repeatable event type (κ = 0.59).The repeatability study used two independent annotations of 1,000 held-out failed ALFWorld trajectories.Source: B ANNOTATION QUALITY · p. 17

Discrimination, density, and rendering

  • State/Action/Result rendering raised Search hop verifications assigned to the retrieving query from 0% to 54% and reduced one-turn-late ALFWorld credit from 39.6% to 0.7%.On 7B ALFWorld, the final potential separated successful from failed episodes with AUC = 0.93, but the outcome-aware annotator makes this a consistency check, not independent validation.Source: B ANNOTATION QUALITY · p. 17

Distilled annotator

  • 92.2% peak success for the distilled Qwen3-8B annotator versus 89.1% for the API annotator, a difference treated as within decoding noise.Its held-out per-turn credit vector correlates with the API annotator at r = 0.81, against API self-agreement of 0.86.Source: B ANNOTATION QUALITY · pp. 17–18

C ALL EXPERIMENTS AT A GLANCE

The experiments-at-a-glance section reports results across interactive tasks, search QA, tool-assisted mathematics, and tool-free reasoning. Outcomes vary by benchmark, checkpoint, and metric rather than showing a uniform gain on every reported measure.

ALFWorld

  • 92.0% ALFWorld success for DARS versus 88.7% for local GiGPO at 1.5B; at 7B, DARS reaches 98.6% at its peak versus GiGPO’s 97.0% final score.The listed 7B checkpoints differ: DARS is reported at peak and GiGPO at final.Source: C ALL EXPERIMENTS AT A GLANCE · p. 19

WebShop

  • At WebShop 1.5B, DARS κ=1 scores 67.7% success / 85.8 task score versus GiGPO’s 65.9 / 84.9; at 7B, DARS scores 76.2 / 87.9 at peak versus GiGPO’s 77.3 / 85.7.The 7B comparison improves task score but not success; these are peak checkpoints for DARS and GiGPO, respectively.Source: C ALL EXPERIMENTS AT A GLANCE · p. 19

Search-R1

  • 43.1 exact-match QA accuracy for DARS versus 38.7 for local GiGPO on the Search-R1 shared pool at final checkpoints.The table also reports train-validation peaks of 44.7 for DARS and 42.4 for GiGPO.Source: C ALL EXPERIMENTS AT A GLANCE · p. 19

Agentic mathematics (Python)

  • AEPO + DARS scores 60.4, 67.5, and 62.9 mean AIME24/25 at steps 5, 10, and 15, while AEPO scores 63.3, 61.7, and 60.0.These are the reported Python-interpreter mathematics results for Qwen3-8B.Source: C ALL EXPERIMENTS AT A GLANCE · p. 20

Tool-free reasoning

  • 79.50 average points for DARS versus 78.44 for OmniOPD in Qwen3-4B tool-free reasoning across AMC, AIME24, and AIME25.The table reports averages over evaluation cells for the 200-step comparison.Source: C ALL EXPERIMENTS AT A GLANCE · p. 20

D BUDGET-MATCHED LOCAL BASELINES AND LEARNING DYNAMICS

Budget-matched experiments compare DARS with locally trained baselines using shared scripts, budgets, and evaluation harnesses where specified. DARS leads on ALFWorld and WebShop graded score in the reported comparisons, while the WebShop 7B strict-success difference is not statistically significant.

Budget-matched local baselines

  • DARS is ahead of matched GiGPO on ALFWorld at both scales and both 1.5B budgets, and on WebShop graded score at both scales; WebShop 7B strict success is not significantly different.For WebShop 7B, the fixed-budget strict-success difference is −1.0 points (p = 0.07); the reported benefit is in graded score.Source: D BUDGET-MATCHED LOCAL BASELINES AND… · p. 20

ALFWorld 1.5B over 500 steps

ALFWorld per-class results

  • On ALFWorld 1.5B per-class results, DARS improves overall success from 88.7% to 92.0% (+3.3 points), with its largest gains on Pick (+15.7) and Clean (+10.6).DARS reaches perfect success on Pick2, while GiGPO remains stronger on Look, Heat, and Cool.Source: Table 10 · p. 22, D BUDGET-MATCHED LOCAL BASELINES AND… · p. 22

Learning dynamics

  • DARS first reaches 70% ALFWorld validation success after 75 optimizer steps versus 85 for GiGPO, and reaches 90% after 120 versus 145 steps.DARS also reaches 70% in 0.23B rollout tokens versus 0.25B for GiGPO and ends with token entropy 0.24 versus 0.48; Figure 3 presents the learning dynamics.Source: Figure 3 · p. 22, D BUDGET-MATCHED LOCAL BASELINES AND… · p. 22

ALFWorld 7B baselines

  • On ALFWorld 7B, GiGPO leads GRPO and RLOO across all eight tested selections and matched steps, with paired gaps of 9.7–23.4 points.The paired tests use shared evaluation seeds; the reference evaluations preceded the new baseline evaluations by about two weeks, without a contemporaneous drift re-evaluation.Source: D BUDGET-MATCHED LOCAL BASELINES AND… · pp. 22–23, Table 11 · p. 23

WebShop 1.5B GRPO

  • WebShop 1.5B GRPO scores 77.7 ± 0.6% task score at its step-145 peak and 76.6 ± 0.8% at step 150, while success favors the final checkpoint.It was evaluated on a different host, so the paper reports no paired comparison with the local GiGPO and DARS references.Source: D BUDGET-MATCHED LOCAL BASELINES AND… · p. 23

E WEBSHOP: DARS CONTINUATION FROM A TRAINED GIGPO POLICY

The WebShop studies test whether DARS improves trained GiGPO policies and whether its reward helps from scratch at a matched budget. Fresh-seed evaluations show gains over parent policies, while an equally extended control separates DARS’s contribution from additional training.

E WEBSHOP: DARS CONTINUATION FROM A TRAINED GIGPO POLICY

E.1 EQUALLY EXTENDED GIGPO CONTROL

  • Against an equally extended GiGPO continuation at 7B, DARS adds 1.5 points of success beyond the parent’s training gain.At 1.5B, additional GiGPO training accounts for the DARS continuation’s gain over the parent.Source: §E.1 · pp. 24–25, Table 13 · p. 25

E.2 FROM SCRATCH AT THE PUBLISHED 150-STEP BUDGET

  • At the matched 150-step budget from scratch, DARS scores 70.6% success / 86.1% task score versus GiGPO’s 65.6% / 83.7%.The advantage is largest at step 100 and has closed by step 200; the comparison uses one training seed.Source: §E.2 · p. 25, Table 14 · p. 25

F TRAINING PROTOCOL AND HYPER-PARAMETERS

The protocol applies a shared reward and annotation setup across task-specific topologies and learners, with task-specific evaluation and training budgets. The section also reports WebShop sensitivity to clipping, diagnostics on clipped credit, and observed annotation costs.

F TRAINING PROTOCOL AND HYPER-PARAMETERS

  • ALFWorld uses per-object ordered chains, WebShop parallel-AND requirements, and Search-R1 and mathematics chains; training budgets and reward settings vary by track.The protocol also specifies unseen-task evaluation for ALFWorld and WebShop and held-out evaluation for Search-R1.Source: F TRAINING PROTOCOL AND HYPER-PARAMETERS · pp. 25–27, Table 15 · p. 27
  • Search-R1 uses hop-chain credit plus answer-commitment terms, while the mathematics comparison adds DARS’s step reward to ARPO/AEPO arms under shared training settings.The Search-R1 commitment terms address the graph-only failure mode of continuing to search without answering.Source: F TRAINING PROTOCOL AND HYPER-PARAMETERS · pp. 26–27
  • With WebShop 1.5B’s optimizer, rollout budget, and step-150 checkpoint fixed, κ = 0 scores 4.7 points lower in success than κ = 1, while a completion verifier adds 1.9 points.These are configuration comparisons on the same validation set, not a confirmatory evaluation of a frozen recipe.Source: F TRAINING PROTOCOL AND HYPER-PARAMETERS · p. 27, Table 16 · p. 27
  • The deployed κ = 0.3 clipping breaks exact telescoping asymmetrically, so the paper does not claim invariance for the clipped reward.Across three on-policy ALFWorld 1.5B runs, positive clipping excess was 1.1–1.4% of positive credit and showed no upward trend.Source: F TRAINING PROTOCOL AND HYPER-PARAMETERS · p. 28
  • Signed credit avoided the progress-only arm’s rising positive-credit events alongside falling success, and the penalty-only arm’s persistently low success and potential.The earlier ALFWorld 1.5B experiment motivates retaining both signs but does not establish immunity to reward farming.Source: F TRAINING PROTOCOL AND HYPER-PARAMETERS · p. 28

G ANNOTATION COST

  • In observed ALFWorld runs, a distilled local annotator took 506 seconds per step versus 512 seconds for the API annotator, while policy generation accounted for about 75% of step time.These are shared-host observations, not a controlled speed benchmark; the local annotator also removes per-account API rate limits and per-call cost.Source: G ANNOTATION COST · pp. 28–29

H SEARCH-R1 DETAIL

Search-R1 results compare shared-pool and held-out protocols, including a test of graph credit and answer commitment. The mathematics section describes its checkpoint comparison and evaluation variability, while the supplied evidence identifies ceiling performance on several benchmarks.

H SEARCH-R1 DETAIL

  • In the shared-pool diagnostic, DARS is higher than GiGPO in all three decoding groups at final checkpoints (360 DARS steps; 662 GiGPO steps), with accuracies pooled over 700 questions and four draws per group.Source: H SEARCH-R1 DETAIL · p. 29
  • Search-R1’s held-out protocol selects checkpoints on val350 and reports exact-match accuracy on a disjoint, subset-stratified test350 split.All arms train for one epoch; test and validation pools are disjoint from each other and from training questions.Source: H SEARCH-R1 DETAIL · p. 29, Table 18 · p. 29
  • On held-out Search-R1, removing either answer-commitment terms or graph credit loses the gain: commitment-only scores 24.6%, while graph-only eventually scores 0%.The commitment terms prevent the graph-only collapse, while graph credit supplies retrieval behavior absent from commitment-only training.Source: Table 18 · p. 29, H SEARCH-R1 DETAIL · p. 29

I AGENTIC MATHEMATICS: FULL DETAIL

  • For agentic mathematics, the primary metric is the AIME24/25 mean because MATH500, GSM8K, and MATH are at ceiling and vary by less than ±1-point decoding variability.The four-arm design uses Qwen3-8B with a Python interpreter and search disabled.Source: I AGENTIC MATHEMATICS: FULL DETAIL · p. 30, Table 19 · p. 30, Table 20 · p. 30
  • Mathematics scores report each arm’s best checkpoint among common steps 5, 10, and 15; “best” is selected on the reported AIME problems, not an independent test estimate.Four decoding draws per 60 problems yield about ±5-point decoding variation, not a formal confidence interval or training-seed variation.Source: I AGENTIC MATHEMATICS: FULL DETAIL · p. 30

J TOOL-FREE REASONING: FULL ARM TABLE AND DIAGNOSTICS

The tool-free reasoning evidence compares DARS with baselines and matched controls, then tests annotation diagnostics and reward ablations. The results distinguish the roles of step credit, dependency attenuation, and graph topology across task settings.

J TOOL-FREE REASONING: FULL ARM TABLE AND DIAGNOSTICS

K.1 WHAT EACH ABLATION REMOVES

  • The λ = 0 ablation removes attenuation along dependency edges, the scalar control removes the step channel, and the chain variant attenuates every later predicate regardless of dependence.These variants preserve different parts of the annotation and reward design.Source: §K.1 · pp. 32–33

K.2 PROTOCOLS

  • In tool-free mathematics at 1.7B, DARS’s λ = 1 arm beats the matched λ = 0 arm on 7 of 10 checkpoint-host pairs, with a 0.51-point mean paired advantage.The DARS-versus-scalar difference has exact permutation p = 0.002.Source: Table 21 · p. 31, §K.2 · p. 33
  • On ALFWorld, λ = 1 scores 92.0% versus 86.0% for λ = 0 at step 200, and 98.6% versus 95.2% at 7B keep-best checkpoints.The λ = 1 arm leads on all four paired draws in the 1.5B comparison and all 15 paired draws at 7B.Source: §K.2 · p. 33
  • On WebShop 1.5B, the parallel-AND requirement graph scores 63.2% success / 79.5% task score versus 49.0% / 61.1% for a sequential chain.The chain is lower on all four paired draws for both metrics.Source: §K.2 · p. 33
  • In the ALFWorld 1.5B decay sweep, λ = 0.5 and λ = 1 outperform milestone counting at λ = 0, while hard masking at λ = ∞ scores lowest.The sweep uses a different annotator and budget from the matched 200-step comparison, and does not distinguish why soft attenuation beats hard masking.Source: §K.2 · p. 33

K.3 OFFLINE CENSUS: WHICH TRAJECTORIES AN ABLATION TOUCHES

The offline census shows that ablations affect different trajectories across task graph structures, while graph-free prefix credit behaves differently in chain and branching tasks. The paired episodes illustrate local decisions but do not measure additional aggregate gains.

K.3 OFFLINE CENSUS: WHICH TRAJECTORIES AN ABLATION TOUCHES

  • Serializing the graph costs WebShop 14.2 points of success, while ALFWorld’s sequential chains show at most a −2.8% worst-decile credit change.On persisted depth-one WebShop graphs, attenuation changes 8.6% of reward vectors and never removes more than half of a trajectory’s credit; attenuation comparisons favor λ = 1 in direction across the reported cases.Source: §K.3 · p. 34

K.4 A GRAPH-FREE PROCESS REWARD: FIRST-ERROR PREFIX CREDIT

  • First-error prefix credit improves ALFWorld by +3.2 points but lowers WebShop’s graded score by 42 points and collapses Search-R1 to 0% success.The earlier studies are reported as motivation, not matched entries of Table 1.Source: §K.4 · p. 34, Table 26 · p. 34

L RECORDED CREDIT AND PAIRED POLICY EPISODES

  • Paired policy episodes connect recorded training annotations to actions and observations, but evaluation examples contain no recorded DARS shaping credits.The examples describe connections to the training rule without assigning retrospective rewards.Source: L RECORDED CREDIT AND PAIRED POLICY… · p. 35

L.1 RECORDED CREDIT WITHIN COMPLETE TRAINING TRAJECTORIES

Recorded trajectories illustrate how DARS credits requirement completion, lost progress, and recovery across WebShop, Search, and ALFWorld. These selected contrasts show local decisions, not aggregate gains or isolated training effects.

L.1 RECORDED CREDIT WITHIN COMPLETE TRAINING TRAJECTORIES

  • WebShop option clicks each receive +0.125 before purchase, while leaving and reopening another product loses and restores two attributes at −0.3 and +0.3.The first trajectory’s requirements already saturate the potential, so buying receives zero shaping credit.Source: §L.1 · p. 35, Table 27 · p. 35, Table 28 · p. 36

L.2 USING A RETRIEVED FACT IN THE NEXT QUERY

  • In Search, DARS uses the established person to resolve the remaining fact, answering correctly after two searches while GiGPO answers incorrectly after three.An off-topic or repeated query does not automatically receive negative shaping; negative shaping requires loss of credited potential.Source: §L.2 · p. 36, Table 29 · p. 36

L.3 AVOIDING IRRELEVANT WORK WITHIN A SUCCESSFUL EPISODE

  • Both ALFWorld policies succeed on the soapbar task, but DARS uses nine actions versus GiGPO’s fourteen and withholds progress credit from the wrong-object delivery.The action-count difference also includes a later navigation detour, not only the initial pickup.Source: §L.3 · p. 36, Table 30 · p. 37

L.4 COMPLETING THE SECOND REQUIRED OBJECT

  • DARS completes the second pillow task in 13 actions, while GiGPO repeats an unsuccessful pickup until reaching the 50-action limit.Separate graph chains for required copies let completing one leave progress credit available for the other.Source: Table 31 · p. 37, §L.4 · p. 37

L.5 CHECKING THE CONDITION THAT MAKES A CANDIDATE ELIGIBLE

  • DARS verifies Ben Sheets’s birthdate exclusion and answers correctly; GiGPO answers Juan Peña, whose name is absent from its retrieved text.The example is a purposeful additional lookup, not a reduction in search count or exhaustive proof that no other pitcher qualifies.Source: §L.5 · p. 38, Table 32 · p. 39

L.6 SELECTION, ARCHIVE COUNTS, AND INTERPRETATION

  • The examples were selected retrospectively and do not establish aggregate accuracy gains or isolate the dense channel from the full training recipe.The archived contrasts are descriptive, not independent trials or estimates of causal effect.Source: §L.6 · p. 39
Loading 2610.01207v1…