PaperlayerView exact revision on arXiv

Source-linked AI summary

SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning

Dingyi Rong, Yue Shi, Chaofan Ma, Zongrui Wang, Jiezhang Cao, Yixin Cao, Liang He, Ning Liu

arXiv:2610.19397v1cs.AI

TL;DR

Long-horizon agentic RL cannot provide targets for useful behaviors absent from policy rollouts, while multi-turn teacher guidance does not directly train on the oracle-controlled future. SERO executes oracle continuations from policy-visited states and weights downstream actions by behavior-policy compatibility. Across ALFWorld and WebShop, it improves benchmark performance, with full continuations and reachability weighting adding ALFWorld gains.

  • Problem

    Improved credit assignment and multi-turn teacher guidance do not directly provide the full oracle-controlled branch as training targets when useful behaviors are missing from policy rollouts.

    Source: §1 · p. 2

  • Method

    SERO restores a policy-visited state, executes a multi-step oracle continuation in the environment, and uses behavior-policy compatibility with preceding oracle actions to weight downstream targets.

    Source: §1 · pp. 2–3

  • Results

    SERO improves ALFWorld success and WebShop strict success for GraphGPO and HGPO at both scales; on ALFWorld, full continuations outperform root corrections and reachability weighting further improves unseen-task performance.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Executable oracle continuations provide multi-step training targets from policy-visited states, while reachability weighting makes deeper guidance effective as policy compatibility with the oracle branch improves.

    Source: §1 · p. 3

    SERO assumes restorable policy-visited states and an executable training-time oracle, and experiments are limited to ALFWorld and WebShop.

    Source: G LIMITATIONS AND FUTURE WORK · p. 24

Abstract

from arXiv · show

Long-horizon agentic RL is constrained by the futures visited by the current policy. When rollouts fail to reach useful behaviors, improved credit assignment cannot provide supervision for states and actions that were never encountered. On-policy distillation (OPD) introduces teacher guidance on policy-induced states, and recent multi-turn variants improve how this guidance is scheduled, replayed, refined, or validated. However, they do not directly use the oracle-controlled future itself as the training trajectory. We introduce \textbf{State-Rooted Executable Rollout Optimization (SERO)}, which augments on-policy agentic RL with executable oracle continuations. SERO restores a state actually reached by the current policy and lets a training-only oracle continue interacting with the real environment, making the downstream observations and oracle actions themselves available for learning. It then applies \textbf{policy-reachability weighting}, which weights each oracle step by the frozen behavior policy's compatibility with the preceding oracle actions. This allows deeper supervision to become effective as the policy becomes more compatible with the oracle branch. The oracle loss is optimized separately from the base RL estimator and no oracle is used at inference time. We evaluate SERO on two challenging interactive-agent benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld, learning the full executable continuation is substantially more effective than supervising only the root oracle action, while policy-reachability weighting further improves out-of-distribution performance.

1 INTRODUCTION

Long-horizon RL can refine credit for sampled behavior, but cannot supply useful action sequences absent from rollouts; teacher guidance on policy-visited states still often supervises only a correction or intervention outcome. SERO executes multi-step oracle continuations from those states and weights downstream targets by policy compatibility; experiments report improvements across two benchmarks and further ALFWorld gains from full continuations and reachability weighting.

1 INTRODUCTION

  • Improved credit assignment can rank sampled behavior more accurately, but cannot provide a useful action sequence as a positive target when every rollout misses it.This gap is especially important early in training and on difficult tasks, where successful branches may be absent from an entire rollout group.Source: §1 · p. 2
  • Multi-turn OPD extends guidance across steps or validates interventions, but its supervision remains centered on corrections, teacher prefixes, or intervention outcomes rather than the full missing branch.Because an executed correction changes later observations and decisions, omitting downstream transitions leaves long-horizon consequences under-supervised.Source: §1 · p. 2
  • SERO restores a policy-visited state and executes a multi-step oracle continuation, making later decisions under the observations created by preceding oracle actions available as training targets.The oracle is training-only.Source: §1 · pp. 2–3
  • Policy-reachability weighting uses frozen behavior-policy compatibility with preceding oracle actions to down-weight poorly supported prefixes and make deeper guidance effective as compatibility improves.The oracle loss is auxiliary; rewards, advantages, and policy ratios of the base RL algorithm remain unchanged.Source: §1 · pp. 2–3
  • SERO improves ALFWorld success and WebShop task success for GraphGPO and HGPO at both model scales; full continuations outperform root corrections on ALFWorld, with weighting adding gains especially on unseen tasks.Evaluations use Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct.Source: §1 · p. 3

2 RELATED WORK

Related work improves learning from policy-generated experience through finer credit assignment and aligns teacher guidance with learner-induced states. SERO instead uses the executable future created by an oracle correction, while retaining the original on-policy estimator and controlling deeper targets by policy compatibility.

2 RELATED WORK

  • Group-based RL and finer credit-assignment methods improve learning from sampled experience, but cannot provide a missing useful branch when it is absent from every rollout.The methods draw on repeated states, interaction history, or cross-trajectory transition structure to refine credit.Source: §2 · p. 3
  • Imitation and OPD query experts on learner-induced states, with multi-turn methods extending guidance through prefixes, refined trajectories, or short interventions.Related approaches also construct corrective supervision from failed trajectories or expert interaction experience.Source: §2 · p. 3
  • SERO learns the executable future induced by a correction: after restoring a policy-visited state, the oracle continues the branch and its resulting observations and actions are used for training.Source: §2 · p. 4
  • Unlike direct privileged optimization for annotations or turn-level credit, SERO preserves the on-policy estimator and uses policy compatibility with oracle prefixes to control deeper sequential supervision.Interactive oracle actions change the environment and create new states.Source: §2 · p. 4

3 PRELIMINARIES

The preliminaries set up long-horizon agent interaction as sequential decisions with sparse or delayed rewards, then describe group-based RL and the motivation for multi-step oracle supervision. SERO adds an executable oracle branch from policy-visited states alongside the base objective.

3.1 LONG-HORIZON AGENTIC REINFORCEMENT LEARNING

  • In long-horizon tasks, an agent repeatedly acts on environment observations and interaction context, while sparse or delayed rewards make individual action contributions difficult to infer from final outcomes.Training saves a simulator snapshot for each visited state so the decision point can be restored for oracle continuation; snapshots are not policy inputs.Source: §3.1 · p. 4

3.2 GROUP-BASED POLICY OPTIMIZATION AND ORACLE SUPERVISION

  • Group-based RL samples multiple trajectories under a frozen behavior policy and computes group statistics instead of training an explicit critic; finer-credit variants analyze repeated states, history, or trajectory transitions.Source: §3.2 · p. 4
  • Training only on a corrective oracle action misses later decisions because executing that action changes subsequent observations; SERO therefore uses the resulting oracle continuation as multi-step supervision.Source: §3.2 · p. 4
  • SERO restores a selected on-policy state, executes a fresh oracle continuation, and applies prefix-support weights to oracle actions as an auxiliary branch without changing base-method reward statistics, advantages, or importance ratios.The oracle branch is combined with the base RL objective in the policy update.Source: Figure 2 · p. 5

4 PROPOSED METHOD

SERO augments on-policy RL with oracle continuations rooted at policy-visited states, using prefix compatibility to weight downstream supervision. Its auxiliary objective leaves the base RL estimator’s rewards, advantages, and policy ratios unchanged.

4.1 OVERVIEW

  • SERO adds an auxiliary oracle branch to on-policy RL, keeping oracle data out of the base method’s reward statistics, advantages, and importance ratios.The framework is modular across on-policy RL objectives.Source: §4.1 · p. 5

4.2 EXECUTABLE ORACLE CONTINUATION

  • SERO restores a policy-visited decision point in a separate environment copy and executes an oracle continuation from that state through the remaining interaction budget.The on-policy suffix and oracle continuation share the restored root and remaining horizon.Source: §4.2 · pp. 5–6

4.3 POLICY-REACHABILITY WEIGHTING

  • Prefix reachability weights reduce deeper oracle supervision when preceding oracle actions have low compatibility under the frozen behavior policy.The root action receives unit weight; deeper supervision can extend further as compatibility improves.Source: §4.3 · p. 6

4.4 ORACLE CONTINUATION OBJECTIVE

  • A branch-level improvement gate controls the auxiliary contribution by comparing oracle and on-policy suffix returns over the same remaining horizon.The branch must also be valid; for binary success returns, the reward-normalization scale is 1.Source: §4.4 · p. 6
  • The oracle objective combines branch improvement, prefix reachability, and supervised likelihood of oracle actions, while remaining separate from the base RL loss.The oracle and environment-restoration machinery are used only during training and are removed at inference.Source: §4.4 · pp. 6–7

5 EXPERIMENTS

Experiments test whether executable oracle continuations and policy-reachability weighting improve long-horizon interactive RL across two benchmarks, two model sizes, and two RL backbones.

5 EXPERIMENTS

  • The evaluation tests whether executable continuations improve on outcome-based RL and single-step supervision, and whether reachability weighting adds benefit.Benchmarks include ALFWorld and WebShop; models are Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, with the ALFWorld expert planner and a WebShop goal-metadata oracle used during training.Source: §5 · p. 7, §5.1 · p. 7

5.2 MAIN RESULTS

  • Across both model scales and RL backbones, SERO improves ALFWorld and WebShop results; the largest gain is 11.20 points on ALFWorld unseen with 7B GraphGPO.Table 1 reports ALFWorld seen/unseen success and WebShop task score/success, averaged over three testing seeds.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 ABLATION OF ORACLE SUPERVISION

  • With Qwen2.5-1.5B GraphGPO, Full-Flat beats Root-only by 7.03/4.43 points on ALFWorld seen/unseen success and 0.33/0.60 points on WebShop task score/success.Root-only uses the first oracle action; Full-Flat trains the complete continuation with a shared branch-level weight.Source: §5.3 · pp. 8–9, Table 2 · p. 8
  • Compared with Full-Flat, SERO gains 1.04/3.39 points on ALFWorld seen/unseen success and 1.27/0.94 points on WebShop task score/success.On the same frozen oracle bank, SERO reaches 88.02/85.94 seen/unseen success, versus 85.42/84.64 for flat weighting and 87.24/82.55 for depth decay.Source: Table 2 · p. 8, §5.3 · p. 9
  • Performance remains strong for λO ∈ [0.1, 0.5] but drops at λO = 1.0; λO = 0.05 is weaker, particularly on unseen ALFWorld tasks.Figure 3 summarizes sensitivity on ALFWorld and WebShop; λO = 0.1 is the default.Source: §5.3 · p. 9, Figure 3 · p. 9

5.4 POLICY REACHABILITY ACROSS ORACLE DEPTH

  • As training progresses, SERO uses increasingly deeper oracle supervision while avoiding equal weights for every downstream action.The later checkpoint maintains larger prefix reachability weights over a longer portion of the continuation than the early checkpoint.Source: §5.4 · p. 9, Figure 4 · p. 9

6 CONCLUSION

SERO trains on executable oracle continuations from policy-visited states, with reachability weighting to adjust supervision for downstream oracle actions. Across controlled evaluations, full continuations and compatibility-aligned weights outperform root-only or less-aligned alternatives, while gains appear across task families and training stages.

6 CONCLUSION

  • Across both model scales, SERO improves ALFWorld success and WebShop strict success for GraphGPO and HGPO; full continuations outperform isolated corrective actions, and reachability weighting further improves generalization.The conclusion summarizes the evaluated scope and the comparative findings of the ablations.Source: §6 · p. 10

A.1 FIXED-BANK WEIGHTING CONTROL

  • 88.02/85.94 seen/unseen success is SERO’s best controlled-replay result; shuffling its weights drops success to 85.16/80.73, showing that weights must align with oracle-prefix reachability.The fixed-bank variants replay the same roots and continuations from the GraphGPO step-50 checkpoint and are evaluated at step 80.Source: §A.1 · p. 17, Table 3 · p. 17

A.2 MATCHED EXTRA-EXPLORATION CONTROL

  • 95.83/91.67 SERO success exceeds the interaction-matched GraphGPO baseline’s 93.49/90.36 by 2.34/1.31 points on seen/unseen tasks.The extra-exploration baseline uses 15 additional training steps to approximately match SERO’s estimated environment-transition budget.Source: §A.2 · p. 17, Table 4 · p. 17

A.3 EMPIRICAL CHECK OF THE REACHABILITY PROXY

  • The frozen policy’s single-step compatibility score correlates with sampled action-match frequency (Spearman ρ = 0.763), while reachability-weight bins track rising oracle-prefix reproduction frequency.The score ranks prefix compatibility but is not the probability of reproducing the full command prefix.Source: §A.3 · p. 18, Table 5 · p. 18

A.4 CURRENT-POLICY ROOT CONTROL

  • After 30 replay updates, roots from the step-50 policy reach 88.0/85.9 seen/unseen success versus 84.4/83.9 for step-10 roots, though task instances and horizons are not exactly matched.Both banks contain 271 roots, but their oracle-step counts differ.Source: §A.4 · p. 18

A.5 ALFWORLD TASK-TYPE BREAKDOWN

  • SERO’s ALFWorld gains span several task families rather than one category; category-level gains and regressions vary across backbones and model scales.The breakdown covers six task families; GraphGPO’s largest gains differ by scale, while HGPO also shows task-specific trade-offs.Source: §A.5 · pp. 18–19, Table 6 · p. 18

A.6 TRAINING DYNAMICS

  • SERO generally accelerates early- and mid-training ALFWorld improvement, while WebShop curves cross more often and the timing and magnitude of gains vary by setting.The training curves compare GraphGPO and HGPO with and without SERO at both model scales.Source: §A.6 · p. 19, Figure 5 · p. 19

A.7 SHORT-BRIDGE CONTROL

  • Short Bridge improves over Root-only but trails full SERO: on ALFWorld, it reaches 91.67/86.72 versus SERO’s 95.83/91.67 seen/unseen success.On WebShop, Short Bridge reaches 87.60/78.77 task score/success, while full SERO reaches 88.90/79.27.Source: §A.7 · p. 20, Table 7 · p. 20

A.8 HORIZON-STRATIFIED EVALUATION

The horizon-stratified evaluation finds SERO gains on unseen long-plan tasks as well as shorter plans. The notation and interpretation clarify that reachability weights are compatibility proxies over preceding oracle actions, not probabilities of reproducing full command prefixes.

A.8 HORIZON-STRATIFIED EVALUATION

  • On unseen ALFWorld tasks, SERO improves success by 6.23 points for short plans and 3.82 points for long plans, with similar performance on medium plans.Seen-split gains are 8.28 and 6.72 points for short and medium plans, while long-plan success is nearly unchanged.Source: §A.8 · p. 20, Table 8 · p. 20

B NOTATION AND ACTION SUPERVISION

  • The policy’s decision state includes the current observation and interaction context exposed by the agent interface; exact restoration may additionally use simulator state held by the wrapper.The notation table collects terms used in the method and experiments.Source: B NOTATION AND ACTION SUPERVISION · p. 20, Table 9 · p. 21

B.1 LANGUAGE COMMANDS

  • For tokenized commands, the geometric mean defines a length-normalized compatibility score, not a calibrated marginal action probability or the probability of emitting the entire command string.The same serialization and command-token mask are used for behavior-policy scoring and the auxiliary loss; reasoning prefixes are not marginalized.Source: §B.1 · p. 21

B.2 POLICY-REACHABILITY INTERPRETATION

  • The exclusive oracle prefix gives the root target unit weight; lower compatibility at a step reduces supervision only for later oracle actions.For tokenized commands, the weight is a length-normalized compatibility proxy rather than the probability of emitting the full prefix.Source: §B.2 · p. 21

C TRAINING ALGORITHM AND RESTORATION

SERO restores recorded policy-visited roots and executes oracle branches in isolated environments. Base RL statistics remain based on primary rollouts, while stored oracle examples contribute through a separate auxiliary loss.

C.1 ROOT RECORDS AND RESTORATION

  • A root record preserves the task, action prefix, observation, remaining budget, legal-action interface, agent history, and original suffix return; restoration is validated against the original policy-visible prompt and actions.The environment is restored directly when serializable, otherwise the adapter resets the task and replays the recorded prefix; mismatched branches are invalidated.Source: §C.1 · p. 21
  • After restoration, the oracle branch maintains its own environment history, so later observations come from its executed actions rather than the original on-policy suffix.Source: §C.1 · p. 21

C.2 DATA AND GRADIENT SEPARATION

  • Primary on-policy rollouts alone determine base advantages and statistics; selected roots are restored in isolated copies before the oracle continuation is executed and recorded.The algorithm freezes a behavior snapshot and computes a detached branch gate for each oracle branch.Source: §C.2 · p. 22
  • The auxiliary SERO loss uses detached branch gates and reachability weights with command-token losses, then adds the normalized objective to the base RL loss.The algorithm normalizes by attempted roots; implementation tests include zero oracle access during evaluation.Source: §C.2 · p. 22

D TASK PROTOCOLS AND ORACLE BACKENDS

ALFWorld oracle branches use the environment’s expert-plan interface, while WebShop uses a deterministic rule-based oracle grounded in goal metadata. Both protocols restore policy-visited failures and execute oracle actions through the environment interface.

D.1 ALFWORLD

  • For ALFWorld, one failed rollout from each group containing a failure supplies a uniformly sampled pre-action root, and expert-plan commands are checked against admissible actions before execution.The adapter reconstructs the on-policy prefix and verifies the prompt and admissible actions; a valid branch gets a positive gate only if it improves the failed rollout from that root.Source: §D.1 · p. 22
  • ALFWorld base and SERO variants share the task manifest, prompt construction, action interface, and evaluation protocol within each backbone.The ALFWorld oracle gate uses binary task success.Source: §D.1 · pp. 22–23

D.2 WEBSHOP

  • WebShop’s deterministic rule-based oracle uses the target product and required options from goal metadata, executes search, navigation, selection, and purchase through the legal interface, and gates on strict success.Root selection samples a failed rollout and a pre-action turn from groups with at least one failure; all-success groups are skipped.Source: §D.2 · p. 23

E.1 ALFWORLD TRAINING CONFIGURATION

The ALFWorld and WebShop configurations keep base and +SERO comparisons matched within each backbone, with benchmark-specific settings reported separately.

E.1 ALFWORLD TRAINING CONFIGURATION

  • ALFWorld base and +SERO runs use the same task manifests, action interface, and decoding settings within each backbone.Table 10 lists the settings used in matched GraphGPO/SERO and HGPO/SERO comparisons.Source: §E.1 · p. 23, Table 10 · p. 23

E.2 WEBSHOP TRAINING CONFIGURATION

  • WebShop base and +SERO runs keep the same horizon, prompt format, action interface, and final-checkpoint evaluation protocol within each backbone.Table 11 summarizes the backbone-specific settings.Source: §E.2 · p. 23, Table 11 · p. 24

F ABLATION DEFINITIONS

The ablations distinguish root-only supervision, full-continuation training with a shared branch weight, and SERO's step-specific prefix weights; fixed-bank controls equalize branch roots and total supervision mass.

F.1 ROOT-ONLY AND FULL-FLAT

  • Root-only retains only the first oracle target, while Full-Flat trains the full continuation with one shared branch weight.SERO uses the same oracle branch but retains step-specific exclusive-prefix weights.Source: §F.1 · p. 24

F.2 FIXED-BANK CONTROLS

  • In the fixed-bank experiment, Flat, Depth decay, and SERO replay identical cached branches from the same starting checkpoint and match total supervision mass per branch.Their auxiliary objectives differ only in where that mass is assigned along the continuation.Source: §F.2 · p. 24

G LIMITATIONS AND FUTURE WORK

SERO relies on restorable policy-visited states and a training-time executable oracle, and the reported experiments cover ALFWorld and WebShop. The authors identify grounding and scale as open extensions.

Scope and assumptions

  • SERO assumes the training environment can restore policy-visited states and an executable oracle is available during training.Experiments are limited to ALFWorld and WebShop.Source: G LIMITATIONS AND FUTURE WORK · p. 24

Visual grounding

  • Visually grounded embodied settings require grounding objects named in oracle commands across successive observations.The authors point to vision-language grounding, segmentation and localization, dense prediction, and 3D scene representations as relevant tools.Source: G LIMITATIONS AND FUTURE WORK · p. 24

Root sampling

  • Uniform sampling of roots from failed rollouts may under-supervise rare failure types.Prioritized sampling and methods for domain shift and long-tailed data are suggested as possible remedies.Source: G LIMITATIONS AND FUTURE WORK · p. 24

Evaluation scope

Related papers

Loading 2610.19397v1…