PaperlayerView exact revision on arXiv

Source-linked AI summary

SCAD: Structured Credit Assignment and Distillation for Long-Horizon Agents

Shangyang Wu, Shuai Zhao, Ziyue Zhu, Jinyang Wu, Anh Tuan Luu, Haoran Luo

arXiv:2610.03372v1cs.LG

TL;DR

Long-horizon agents need useful supervision across extended sequences, yet terminal outcomes obscure decision-level credit and teacher guidance can lose value along student-generated histories. SCAD combines local execution distillation, cross-rollout planning credit, and differentiated signal allocation. Across benchmarks, it improves macro-average accuracy over the strongest training baseline for both text and multimodal tasks.

  • Problem

    Long-horizon agents need supervision across extended interaction sequences, but terminal outcomes obscure individual contributions and teacher feedback can lose discriminative value along student-generated histories.

    Source: §1 · pp. 1–2

  • Method

    SCAD distills bounded execution in local contexts, estimates planning credit from cross-rollout subtask–report prefixes, and assigns planning and execution distinct reward and teacher signals.

    Source: §1 · p. 2

  • Results

    SCAD improves macro-average accuracy over the strongest trained baselines by 4.48 points on text tasks and 4.19 points on multimodal tasks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    SCAD achieves these gains without process rewards, a learned critic, or extra counterfactual branch sampling.

    Source: §5 · p. 9

    Evaluation is limited to compact Qwen students on text and image-based search tasks.

    Source: H LIMITATIONS AND FUTURE WORK · p. 32

Abstract

from arXiv · show

Training long-horizon agents to solve complex tasks requires effective supervision over extended interaction sequences. However, sparse terminal rewards obscure intermediate contributions, while on-policy distillation can lose informative teacher guidance as student-generated histories grow. To address this problem, we introduce SCAD, which organizes interactions into planning and bounded subtask execution, distills execution in local contexts, and refines planning credit through cross-rollout subtask prefix trees, with planning receiving full terminal credit and execution receiving positive terminal credit and teacher guidance. Across all evaluated benchmarks, SCAD improves macro-average accuracy over the strongest training baseline by 4.48 percentage points for text tasks and 4.19 points for multimodal tasks. SCAD effectively combines outcome-based credit assignment with teacher-guided distillation to improve planning and execution in long-horizon agents.

1 INTRODUCTION

Long-horizon training needs supervision across extended sequences, but long histories can weaken teacher feedback and obscure credit for individual decisions. SCAD combines local execution distillation, cross-rollout planning credit, and differentiated signal allocation.

1 INTRODUCTION

  • Prior approaches organize supervision at different scales, including bounded execution contexts, planning–execution advantages, shared-state credit, tree-search rollouts, and turn-level distillation.Examples include Context Folding and CoDA, HiPER and GiGPO, Tree-GRPO, and TCOD (WANG et al., 2026).Source: §1 · p. 1
  • SCAD addresses long-horizon training challenges: teacher feedback can lose discriminative value, terminal outcomes obscure individual decision contributions, and negative outcomes can suppress useful execution guidance.The paper identifies these problems as maintaining useful teacher supervision, assigning credit under long-range dependencies, and coordinating task-level rewards with local learning.Source: §1 · p. 2
  • SCAD distills bounded subtask execution in local contexts, estimates planning credit from matched subtask–report prefixes across completed rollouts, and allocates signed terminal credit to planning but only positive terminal credit plus teacher guidance to execution.This allocation preserves teacher guidance in failed trajectories and reinforces execution associated with task success.Source: §1 · p. 2
  • SCAD improves overall macro-average accuracy over the strongest training baseline by 4.48 points for text and 4.19 points for multimodal benchmarks.Figure 2 presents benchmark accuracy across text and multimodal tasks, with SCAD highlighted in red.Source: §1 · p. 2, Figure 2 · p. 2

2 RELATED WORK

Prior work addresses long-horizon agents through architectural context management and varied credit-assignment and distillation methods. SCAD combines local teacher contexts with subtask–report prefixes for planning credit across completed rollouts.

2 RELATED WORK

  • Long-horizon architectures separate planning and execution, bound execution contexts, or compress histories; SCAD applies context isolation to teacher supervision while retaining task history for planning.Examples include AgentFlow (Li et al., 2026e), Context Folding and CoDA, and AgentFold and IterResearch.Source: §2 · p. 2
  • Prior credit-assignment methods use return redistribution, intermediate rollouts, shared-state or prefix comparisons, and hierarchical advantages, while distillation methods include on-policy learning, temporal curricula, and memory alignment.The cited examples include Arjona-Medina et al. (2019), Kazemnejad et al. (2025), Feng et al. (2025), Peng et al. (2026), Agarwal et al. (2024), and Liu et al. (2026c).Source: §2 · p. 2
  • SCAD distinguishes its supervision by pairing subtask-bounded local teacher contexts with subtask–report prefixes for planning credit, while assigning signed terminal credit to planning and its positive part plus teacher guidance to execution.Table 1 compares how representative methods organize supervision for long-horizon agents.Source: Table 1 · p. 3

3 METHODOLOGY

SCAD formulates long-horizon interaction as trajectories judged by terminal success, then separates planning from bounded subtask execution. It combines local teacher supervision with cross-rollout planning credit and allocates the resulting signals by token role.

3 METHODOLOGY

  • Tasks are evaluated by a binary terminal reward that indicates whether a completed trajectory succeeds against its reference outcome.The policy generates actions from task input and interaction history; a frozen teacher supplies intermediate feedback, while terminal success remains the objective.Source: §3 · p. 3

3.1 SUBTASK-LOCAL ON-POLICY DISTILLATION

  • Execution handles bounded subtasks in local contexts, while planning delegates or finishes and receives only execution reports in its accumulated history.Teacher scores student-sampled execution tokens in the same local context; planning tokens receive no teacher supervision, and the local transcript budget reserves space for the report.Source: §3.1 · pp. 3–4

3.2 CROSS-ROLLOUT PLANNING CREDIT

  • Across completed rollouts, SCAD matches clustered subtasks and canonicalized reports into prefix trees, then compares terminal-return action values under shared histories.A prior shrinks the parent baseline toward the problem-level mean, with less influence as visits grow; the scaled tree advantage is capped and applies only to planning spans, while reports match histories rather than verify subtask outcomes.Source: §3.2 · pp. 4–5

3.3 STRUCTURED SIGNAL ALLOCATION AND POLICY OPTIMIZATION

  • Planning tokens combine terminal and tree credit, while execution tokens combine positive terminal credit with signed local teacher feedback through role-specific allocation.When group outcomes agree, terminal credit vanishes but distillation remains; the shared policy is optimized with a token-averaged clipped objective using fixed rollout probabilities, teacher scores, and advantages.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate SCAD across text and multimodal long-horizon benchmarks and analyze its components, supervision, planning credit, signal allocation, and efficiency. SCAD achieves higher overall macro accuracy than the strongest trained baselines in both modalities.

4.1 EXPERIMENTAL SETUP

  • Evaluation uses 814 text and 510 multimodal questions, with final-answer accuracy judged by Gemini-3.7-Flash and dataset accuracies equally weighted within each modality.Training uses 400 text and 347 multimodal examples; RQ3–RQ6 use text diagnostics.Source: §4.1 · p. 6, §4 · p. 6

4.2 MAIN RESULTS (RQ1)

  • SCAD reaches 46.10% text and 33.13% multimodal overall macro accuracy, exceeding ATOD at 41.62% and HyperEyes at 28.94% by 4.48 and 4.19 points.It leads trained baselines across individual datasets; the multimodal student exceeds its teacher by 2.23 points in overall accuracy.Source: §4.2 · p. 7

4.3 ABLATION STUDY (RQ2)

  • Local OPD adds 3.87/9.46 points, tree credit adds 1.24/1.61, and structured allocation adds 1.11/2.09 to text/multimodal accuracy, reaching 46.10/33.13%.Local OPD accounts for 62.2/71.9% of the total gain in this sequence; tree credit and allocation jointly add 2.35/3.70 points.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4 ANALYSIS OF SUBTASK-LOCAL DISTILLATION (RQ3)

  • At depth 24, teacher correct-versus-incorrect margins are 0.126 nats/token in local contexts and 0.039 in full-history contexts; local OPD + GRPO reaches 43.75%, 2.51 points above ATOD.The continuation accuracy difference falls from 7.82 to 1.28 points across depths 1/8/16/24; continuation is diagnostic only.Source: §4.4 · p. 8

4.5 ANALYSIS OF CROSS-ROLLOUT PLANNING CREDIT (RQ4)

  • Full tree credit reaches 46.10% accuracy, 1.49 points above uniform scaling, while multi-candidate groups cover 23.51% of planning visits and occur in 72.66% of trajectories.The comparison holds Worker supervision fixed and changes only Planner credit.Source: §4.5 · p. 8, Figure 7 · p. 8

4.6 ANALYSIS OF STRUCTURED SIGNAL ALLOCATION (RQ5)

  • With planning fixed, positive-only execution credit raises accuracy from 44.99% to 46.10%, while subtask and terminal labels disagree on 26.49% of subtasks.Positive GRPO + OPD and SCAD improve correct/failed retention by 3.51/3.64 points.Source: §4.6 · p. 9, Figure 8 · p. 9

4.7 TOKEN AND COMPUTATIONAL EFFICIENCY (RQ6)

  • At 24 million teacher-supervised tokens, SCAD reaches 43.51% accuracy versus ATOD’s 41.53%; over 60 steps, SCAD takes 4.92 hours versus ATOD’s 7.93.The token budget counts teacher-scored response tokens for OPD, not all student rollout tokens; SCAD uses 37.96% less time than ATOD.Source: §4.7 · p. 9, Figure 9 · p. 9

5 CONCLUSION

The conclusion presents SCAD as a combination of local execution distillation, prefix-tree planning credit, and structured signal allocation. It reports macro-accuracy gains over the strongest trained baselines and identifies adaptive teacher allocation as future work.

5 CONCLUSION

  • SCAD combines subtask-local distillation, prefix-tree planning credit, and structured signal allocation, with full terminal credit for planning and positive terminal credit plus local teacher feedback for execution.The method uses no process rewards, learned critic, or extra counterfactual branch sampling.Source: §5 · p. 9
  • Across benchmarks, SCAD improves macro accuracy over the strongest trained baselines by 4.48 points on text tasks and 4.19 on multimodal tasks.Source: §5 · p. 9
  • Future work will explore adaptive allocation of teacher supervision across subtasks, guided by difficulty and expected learning benefit within a fixed teacher computation budget.Source: §5 · p. 9

A.1 ADDITIONAL STUDENT EXPERIMENTS

This section examines SCAD in the smaller-student setting, including text accuracy and planning-credit variants, and documents the role-separated prompts and runtime protocol. It also presents the conditional teacher-only update’s descent analysis and the limits of its guarantee.

A.1 ADDITIONAL STUDENT EXPERIMENTS

  • 39.58% overall accuracy lets SCAD exceed ATOD’s 35.32% and TCOD’s 35.04%, leading all eleven student columns.It also scores 42.50% on WebWalkerQA, 47.18% on 2Wiki, and 36.00% on xbench.Source: Table 4 · p. 16, §A.1 · p. 16
  • 11.79 points of accuracy gain take SCAD from 27.79% to 39.58%; it overtakes ATOD around step 30 and retains its lead.SCAD reaches 43.68% ID and 32.42% OOD macro accuracy; the 1.7B student retains 85.86% of the 4B student’s overall accuracy using 42.5% of its nominal parameters.Source: Figure 10 · p. 16, §A.1 · p. 16

A.2 COVERAGE AND EFFECTIVENESS OF PLANNING CREDIT

  • 46.10% accuracy results from combining sibling contrast and shrinkage, a 1.78-point gain over no tree credit.Multi-candidate parents occur in 72.66% of trajectories, and 47.17% of their parent–action groups pool returns from at least two rollouts; changing ν shifts siblings equally before clipping while preserving pairwise differences.Source: Table 5 · p. 17, §A.2 · p. 17, Figure 11 · p. 17

A.3 CLUSTERING AND PRIOR-COUNT SENSITIVITY

  • Accuracy spans 45.31–46.10% across wider threshold ranges and 45.68–46.10% for ν ∈ {1, 2, 4, 8}; larger ν increases credit magnitude without monotonic accuracy gain.The sensitivity study varies one parameter at a time around (τs, τr, ν) = (0.83, 0.90, 2).Source: Table 6 · p. 17, §A.3 · p. 17

B.1 PLANNING PROMPTS

  • The Planner delegates self-contained research subtasks, integrates Worker reports, and submits the answer, while Workers execute only their assignments and return evidence.Planning and execution receive distinct role prompts and tool permissions; planning cannot search directly, and Workers cannot delegate or finish the original question.Source: §B.1 · p. 18, §B.2 · p. 19

B.2 LOCAL EXECUTION PROMPTS

  • Local execution starts from the original question and current assignment, excluding generated planning history and earlier execution transcripts.Teacher and student score generated execution tokens under the same local context; budget controls reserve room for a final report, and multimodal resets preserve original images.Source: §B.2 · p. 19

B.3 SHARED TOOL-CALLING PROTOCOL

  • The shared protocol serializes tool calls, while multimodal prompts retain original images and runtime reminders enforce assignments, request reports, and hand results back to planning.Image paths identify attached inputs rather than replacing pixels, and new Worker contexts receive the original images while only returned text reports enter planning history.Source: §B.3 · p. 20, §B.4 · p. 20, §B.5 · p. 21

B.6 FINAL-ANSWER EVALUATION PROMPT

  • The terminal evaluator supplies binary task reward from answer-reference comparison, separately from the frozen teacher’s execution-token scoring.Text tasks may receive normalized answer matches before judge evaluation; multimodal evaluation judges nonempty answers directly, and SCAD disables the optional subtask-scope judge.Source: §B.6 · p. 22

C.1 PROOF OF PROPOSITION 1

  • At the rollout policy, g_b is the conditional teacher-only surrogate gradient, with deviation from reverse-KL descent bounded by Lη_b and zero when no gap is clipped.At θ0, PPO and dual clipping are inactive for ϵ_c > 0 and c_d > 1.Source: §C.1 · p. 23
  • The bound is a strict descent guarantee when ∥∇K_E(θ0)∥ > Lη_b, and does not require independent preceding subtasks.It applies to fixed contexts and unfiltered on-policy tokens; length filtering and the combined task update need not preserve descent, while earlier decisions can still affect assignments and observations.Source: §C.1 · p. 23

C.2 PROOF OF PROPOSITION 2

The proof decomposes tree credit into within-choice variance removal and prior shrinkage, showing that shrinkage preserves sibling contrasts while clipping can alter mean identities.

C.2 PROOF OF PROPOSITION 2

  • Averaging returns by choice removes exactly the within-choice return variation from the empirical variance.At a visited prefix, records are sampled uniformly and choices are weighted by visit count.Source: §C.2 · p. 23
  • Prior shrinkage changes the mean but leaves sibling contrasts and variance unchanged.With one choice, credit is solely the prior offset; multiple choices also contribute between-choice variation.Source: §C.2 · p. 24
  • The exact contrast and mean identities hold before clipping, which can merge advantages and alter their mean.Source: §C.2 · p. 24

C.3 PROOF OF PROPOSITION 3

SCAD’s proof establishes that its execution-credit rule preserves teacher guidance under negative terminal credit. The appendix then describes rollout collection, teacher scoring, and prefix-tree planning credit.

C.3 PROOF OF PROPOSITION 3

  • SCAD’s execution advantage has a derivative at least as large as uniform fusion away from clipping kinks, and failed rollouts retain teacher-only credit.The derivative comparison uses the same probability ratio; when the terminal return is zero, the surrogate equals the teacher-only term across clipped regions.Source: §C.3 · p. 24
  • Shared parameters and normalization prevent the proof from guaranteeing that a larger scalar coefficient raises token probability or monotonically improves task performance.The guarantee is about the direct teacher signal, not isolated behavior or task-level improvement.Source: §C.3 · p. 24

D SCAD ALGORITHM DETAILS

  • The algorithm collects planning and execution sequences, scores local execution with a teacher, builds a subtask–report prefix tree for planning credit, and updates one shared policy.The same procedure supports text-only and image-containing inputs.Source: D SCAD ALGORITHM DETAILS · p. 25

D.1 LOCAL ROLLOUT COLLECTION AND TEACHER SCORING

  • Teacher scoring is applied to stored execution sequences in their local contexts, while planning receives no distillation signal.The three algorithm modules separate rollout collection and teacher scoring, prefix-tree planning credit, and structured policy updates.Source: D SCAD ALGORITHM DETAILS · p. 25

D.2 PREFIX CONSTRUCTION AND PLANNING CREDIT

  • Prefix-tree planning credit compares terminal returns among choices under shared histories, with prior shrinkage and fallback scales addressing sparse prefixes.The appendix describes action values grouped by subtask choice under a parent and uses the input-level return statistics when local observations are sparse.Source: §D.2 · p. 26

D.3 SIGNAL ALLOCATION AND END-TO-END TRAINING

SCAD allocates different learning signals to planning and execution, then trains the shared policy using grouped rollouts. The benchmarks span text and multimodal datasets, including in- and out-of-distribution questions.

D.3 SIGNAL ALLOCATION AND END-TO-END TRAINING

  • Planning receives signed outcome and tree credit, while execution receives positive outcome credit and the signed teacher gap.Teacher, encoder, contexts, and advantages stay fixed during each update; equal rollout returns leave execution-side distillation intact.Source: §D.3 · p. 27

E BENCHMARK DETAIL

  • The benchmark comprises 19 datasets and 1,324 test questions: 814 text and 510 multimodal, with seven OOD datasets absent from training.Bamboogle, PopQA, and FVQA contribute validation examples; GAIA, xbench, InfoSeek, and LiveVQA are evaluation-only.Source: E BENCHMARK DETAIL · p. 28

E.1 DATASET DETAILS

  • Text and multimodal datasets cover fact-finding, multi-step evidence composition, web traversal, visual-textual search, and cross-modal verification.The OOD sets include text research and factual tasks, plus visual entity and news-image questions.Source: §E.1 · p. 28

E.2 EVALUATION DETAILS

Evaluation uses final-answer accuracy and equal-dataset macro averages, with shared data, tools, models, and protocols for baseline comparisons. The appendix specifies reference, reinforcement-learning, and supervised-training baselines.

E.2 EVALUATION DETAILS

  • Final-answer accuracy uses binary correctness, and macro averages weight datasets equally, with text, multimodal ID, and multimodal all-dataset averages defined separately.Text averages 11 datasets; multimodal ID averages five, and multimodal All averages eight.Source: §E.2 · p. 28

F BASELINE DETAIL

  • Baseline methods share models, training pools, tools, and evaluation protocols while retaining their own context management and learning objectives.The appendix provides search implementations and baseline hyperparameters for the comparison tables.Source: F BASELINE DETAIL · p. 29

F.1 SHARED COMPARISON SETTINGS

  • Shared comparison settings specify student and teacher models, training pools, tools, interaction budgets, optimization, and validation-based checkpoint selection.Three seeds share splits, and the reported results use mean and sample standard deviation.Source: §F.1 · p. 29

F.2 REFERENCE MODELS AND SUPERVISED FINE-TUNING

  • Reference methods include direct-answer Vanilla and teacher models, while SFT trains on fixed teacher trajectories using response-token cross-entropy.Vanilla has no task-specific updates; the teacher’s table row reports generated final-answer accuracy.Source: §F.2 · p. 29

F.3 REINFORCEMENT-LEARNING BASELINES

  • RL baselines range from flat terminal-reward GRPO and folded subtask execution to tree rollouts and hierarchical planning and execution advantages.FoldGRPO and Tree-GRPO use signed terminal credit; HiPER uses a value head to estimate level-specific baselines.Source: §F.3 · p. 30

F.4 ON-POLICY DISTILLATION AND HYBRID METHODS

The appendix details on-policy distillation and hybrid baselines, multimodal search methods, and controls for comparing SCAD’s components and efficiency. These methods retain distinct supervision signals, search structures, or credit assignments.

F.4 ON-POLICY DISTILLATION AND HYBRID METHODS

  • OPD, SDAR, ATOD, TCOD, and MemOPD vary teacher inputs, history handling, curriculum, and the combination of teacher guidance with reward signals.OPD uses teacher-scored sampled tokens; SDAR uses privileged-context skill summaries, while TCOD expands the student-controlled prefix and MemOPD reconstructs invocation context after memory rewriting.Source: §F.4 · p. 30

F.5 MULTIMODAL SEARCH AND DISTILLATION

  • Multimodal baselines use multi-turn visual retrieval, terminal-reward search, joint GRPO–OPD, or parallel grounded queries; the online comparison shares the student, training pool, and update count.VOLD applies distillation to unsuccessful trajectories, while HyperEyes supplies OPD on failed rollouts.Source: §F.5 · p. 31

F.6 SCAD AND CONTROLLED COMPARISONS

  • Controlled comparisons add local OPD, tree credit, and structured allocation in stages, vary planning-credit and execution-signal assignments, and measure teacher-supervised tokens and training time.The controls distinguish Planner–Worker comparisons from flat GRPO + OPD, and the timing comparison excludes answer judging.Source: §F.6 · p. 31

G CASE STUDY

The case study traces three dependent subtasks from identifying Will Rogers to verifying a film connection; the Planner uses each report to guide subsequent instructions.

G CASE STUDY

  • The first report identifies Will Rogers but leaves his son’s series unresolved and includes an unverified 1952 title.Source: G CASE STUDY · p. 32
  • Will Rogers Jr.’s 1953–1954 radio series is identified as Rogers of the Gazette, a drama, and linked to The Story of Will Rogers (1952).Source: G CASE STUDY · p. 32
  • The final branch verifies The Story of Will Rogers as the film preceding Rogers of the Gazette before the Planner submits its complete answer.Table 7 presents three dependent subtasks and reports, with terminal reward R = 1.Source: G CASE STUDY · p. 32, Table 7 · p. 32
  • The Planner reads each report and incorporates relevant names into later subtask instructions, linking separate local executions.Source: G CASE STUDY · p. 32

H LIMITATIONS AND FUTURE WORK

SCAD’s evaluation and design have defined scope and comparison constraints, while the authors identify broader evaluation and adaptive allocation as future work.

H LIMITATIONS AND FUTURE WORK

  • SCAD is evaluated with compact Qwen students on text and image-based search tasks.Source: H LIMITATIONS AND FUTURE WORK · p. 32
  • Local distillation depends on teacher competence, while prefix-tree credit depends on history matching and observed alternatives; sparse sibling comparisons use prior shrinkage.Source: H LIMITATIONS AND FUTURE WORK · p. 32
  • Worker budgets and signal weights remain fixed across tasks; future work includes broader tasks, adaptive sampling and matching, and allocation by subtask difficulty and learning benefit.Source: H LIMITATIONS AND FUTURE WORK · p. 32

Related papers

Loading 2610.03372v1…