PaperlayerView exact revision on arXiv

Source-linked AI summary

Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning

Yingjian Chen, Lang Zhou, Zixin Zhao, Pingfan Li, Weishi Zheng, Zhilin Zhao

arXiv:2610.37195v1cs.LGcs.AI

TL;DR

Long-horizon RL must assign feedback from interaction sequences to individual actions, while local comparisons such as GiGPO still use each action’s single sampled continuation. ConCal recursively calibrates continuation values by combining shared evidence from matched visits with each visit’s own transition estimate. It improves success rates over GiGPO on ALFWorld and WebShop, while its matching rule can introduce bias when visits differ in unmodeled history or interaction budget.

  • Problem

    Long-horizon RL must translate feedback from full interaction sequences into credit for individual decisions, while local action estimates can depend on a single sampled continuation.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Method

    ConCal matches visits by incoming state-action-state transitions, blends shared continuation evidence with each visit’s transition estimate, and propagates calibrated values backward without additional rollouts.

    Source: §1 · p. 2

  • Results

    ConCal improves success rates over GiGPO by 3.39 pp on ALFWorld and 5.86 pp on WebShop.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    In the evaluated settings, shared continuation evidence calibrates relative action-value estimates, with analyses reporting improved agreement with held-out action orderings.

    Source: §6 · p. 9

    Matching uses observed incoming transitions without enforcing agreement in full history, hidden state, or remaining interaction budget, which can bias shared estimates.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement learning for long-horizon agents relies on feedback from sequences of interactions, making credit assignment to individual actions difficult. Existing methods such as GiGPO compare actions at shared states to compute relative advantages, but estimate each action's value from a single sampled continuation. The resulting local credit remains sensitive to decisions made later in the trajectory. To address this problem, we propose ConCal (Continuation Calibration), a recursive continuation-value estimator that incorporates continuation evidence already present in a rollout batch. ConCal matches visits from the same task by their incoming state-action-state transitions, combines shared continuation evidence with each visit's own transition estimate, and propagates calibrated values backward to refine earlier action-value estimates. This reduces dependence on individual sampled continuations and mitigates local credit distortions without additional rollouts. Held-out return diagnostics show lower error in within-group value differences and higher agreement with reference action orderings than estimates based on individual continuations. With Qwen2.5-1.5B-Instruct, ConCal improves success rates over GiGPO by 3.39 and 5.86 percentage points on ALFWorld and WebShop, respectively. Our code is available at https://github.com/Yingjian-Chen/ConCal.

1 Introduction

Long-horizon RL makes it difficult to assign task-level feedback to individual decisions, and local comparisons still rely on each action’s sampled continuation. ConCal recursively combines shared continuation evidence with each visit’s own transition estimate; the paper reports improved benchmark success rates over GiGPO.

1 Introduction

  • Long-horizon RL must translate feedback from full interactions into credit for individual decisions.Existing local comparisons still estimate each action from its own sampled continuation, leaving values sensitive to later decisions.Source: §1 · pp. 1–2
  • ConCal matches same-task visits by incoming state-action-state transitions, then recursively blends shared continuation evidence with each visit’s own transition estimate.It propagates calibrated values backward and evaluates actions using their own observed reward and actual successor, without additional rollouts.Source: §1 · p. 2
  • Held-out Figure 2 diagnostics show lower relative-value error and greater local credit consistency than estimates based on individual continuations.The figure describes credit-conflict rates and centered-value MSE on WebShop and ALFWorld rollout subsets.Source: §1 · p. 2, Figure 2 · p. 2
  • With Qwen2.5-1.5B-Instruct, ConCal improves success rates over GiGPO by 3.39 pp on ALFWorld and 5.86 pp on WebShop.The introduction also reports higher mean success rates than GraphGPO on both benchmarks.Source: §1 · p. 2

2 Related Work

Related work spans reinforcement learning for interactive language agents, group-based local credit assignment, and methods that reuse structure across trajectories. This paper focuses on calibrating the action-value estimates used in local comparisons with shared continuation evidence.

Reinforcement learning for language agents

  • Multi-turn RL trains sequential decision-making from environment feedback, with applications including device control, web navigation, and search-augmented reasoning.Recent frameworks support long-horizon training across interactive environments.Source: §2 · pp. 2–3

Group-based local credit assignment

  • Group-based RL estimates relative advantages from sampled rewards without a learned value baseline, while later work develops local comparisons using shared state anchors or historical context.The paper locates its focus in calibrating values entering these comparisons.Source: §2 · p. 3

Cross-trajectory credit estimation

  • Cross-trajectory methods reuse shared structure through tree-based comparisons, transition averaging, graph-derived values or distances, and grouped state potentials.ConCal instead recursively combines shared continuation evidence with visit-specific transition estimates.Source: §2 · p. 3

3 Preliminaries

The preliminaries define task-specific rollout groups and the episode-level and local signals used in group-based agent RL. They then frame local action-value estimation as an expectation over future returns and motivate an estimator that reuses continuation evidence across trajectories.

3.1 Group-Based Agent RL

  • Each task produces a rollout group of trajectories, and the full training batch contains groups for all sampled tasks.Each trajectory consists of up to H transitions, with decision steps treated as visits to observed states.Source: §3.1 · p. 3
  • GRPO assigns every action in a trajectory the same episode-relative advantage, while the combined training signal also includes normalized local advantages and drives PPO-style optimization.Episode advantages compare trajectory returns within a rollout group.Source: §3.1 · p. 3

3.2 Local Action-Value Estimation

  • An action value is the expected discounted return over the remaining episode under the rollout policy, conditioned on the pre-action history and action.That expectation includes subsequent policy decisions and environment transitions.Source: §3.2 · pp. 3–4
  • GiGPO’s local comparison uses each action’s single sampled return, motivating an estimator that incorporates continuation evidence from matched visits without additional rollouts.The target is to improve local action-value estimates while retaining evidence already present across trajectories.Source: §3.2 · p. 4

4 Continuation Calibration

ConCal is a recursive continuation-value estimator that shares evidence across matched visits, calibrates continuation values, and uses them to produce action values and relative advantages for policy updates.

4.1 Matching Visits

  • ConCal matches visits by task and incoming state-action-state transition, allowing shared continuation evidence even when visits occur at different timesteps.Source: §4.1 · p. 4

4.2 Recursive Continuation Calibration

  • At each recursion depth, ConCal blends each visit’s transition estimate with the matched-pool mean while retaining a visit-specific estimate.The shared-mean weight increases with the number of distinct contributing trajectories.Source: §4.2 · pp. 4–5
  • Recursive updates use successor estimates from the previous depth and propagate shared continuation evidence backward through recorded transitions.Source: §4.2 · p. 5

4.3 Action Values and Policy Optimization

  • Calibrated continuation values produce action values and step advantages, which are combined with episode-level advantages for policy optimization.The action retains its observed reward and actual successor; the combined advantages enter a clipped objective.Source: §4.3 · p. 5

5 Experiments

Experiments evaluate ConCal on ALFWorld and WebShop, comparing it with existing methods and testing value calibration, component choices, normalization, and continuation reuse. Results cover benchmark performance, held-out action-value diagnostics, computational cost, and a WebShop case study.

5.1 Experimental Setup

  • The evaluation uses ALFWorld household tasks and WebShop product-purchasing tasks, training Qwen2.5-1.5B-Instruct for 150 updates with eight trajectories for each of 16 tasks per update.Reported main results are means and sample standard deviations across three independent training runs.Source: §5.1 · p. 6

5.2 Experimental Results

  • ConCal records the highest reported aggregate mean success rates among compared methods: 94.27% on ALFWorld and 79.69% on WebShop, with a WebShop task score of 90.18.Relative to GiGPO, success rates rise by 3.39 pp on ALFWorld and 5.86 pp on WebShop; WebShop task score rises by 2.24 points.Source: §5.2 · pp. 6–7
  • On ALFWorld, ConCal improves faster during early-to-middle training; on WebShop, its success-rate advantage over GiGPO appears later and persists through the final updates.The gains emerge at different training stages under the reported rollout schedule.Source: Figure 4 · p. 6, §5.2 · p. 7

5.3 Quality of Local Action-Value Estimates

  • ConCal reduces centered-value MSE by 34.3% on ALFWorld and 16.8% on WebShop, while ordering agreement rises from 58.8% to 61.2% and from 74.5% to 81.8%, respectively.Gap-matched rescales Own-MC values without changing their ordering and reproduces most of the MSE reduction, indicating that preference-strength adjustment accounts for much of the error reduction.Source: §5.3 · p. 7, Figure 5 · p. 7

5.4 Component Analysis; 5.4.1 Continuation Estimation

  • Shared-evidence estimators outperform Own-MC across reported metrics; ConCal exceeds Single-pass in success rates and Fully pooled across all metrics.Single-pass has a slightly higher mean WebShop task score, while ConCal varies less across seeds; Fully pooled removes each visit’s separate contribution.Source: §5.4.1 · pp. 7–8

5.4.2 Advantage Normalization Diagnostic

  • Group-wise normalization yields similar median advantage magnitudes across value-spread bins, whereas batch-wise normalization increases median magnitudes with spread and preserves calibrated preference-strength differences.The comparison uses frozen ConCal estimates grouped into five bins from smaller to larger centered-value gaps.Source: §5.4.2 · p. 8, Figure 6 · p. 8

5.5 Continuation Reuse and Computational Cost

  • Recursive propagation extends shared-continuation coverage from 55.4% to 92.5% on ALFWorld and from 43.4% to 67.5% on WebShop, with matching and calibration taking 0.0375 s and 0.0019 s per batch.The reported matching and calibration costs are small relative to rollout at 108.85 s and policy updates at 19.00 s, under the stated measurement setup.Source: §5.5 · pp. 8–9, Figure 7 · p. 8

5.6 Case study

  • In a WebShop case, ConCal reverses GiGPO’s preference between selecting required size 5X and repeating Zest, changing the value gap from −6.9834 to +0.6452.The sampled continuation for selecting 5X fails, but shared continuation evidence leads ConCal to assign that action positive step advantage.Source: §5.6 · p. 9, Figure 8 · p. 9

6 Conclusion

ConCal reuses shared continuation evidence within rollout batches to refine action values, and experiments report improved task performance and action-order agreement in the evaluated settings.

6 Conclusion

  • ConCal reuses matched-visit evidence and recursively propagates calibrated continuation values backward without additional rollouts.Ablations support recursive evidence reuse and visit-specific estimation.Source: §6 · p. 9
  • Experiments on ALFWorld and WebShop show improved task performance.The conclusion reports the result in those evaluated settings.Source: §6 · p. 9
  • Further analyses find that shared continuations calibrate preference strength and improve agreement with action orderings derived from held-out returns.The paper supports calibrating relative action-value estimates with shared continuation evidence from existing rollout groups in the evaluated settings.Source: §6 · p. 9

Limitations

ConCal matches visits using observed incoming transitions, a rule that may not capture other aspects of interaction context and whose reuse depends on matching visits being available.

Limitations

  • Matching requires the same observed incoming transition but does not enforce agreement in full interaction history, hidden state, or remaining interaction budget.These differences can bias shared estimates incorporated into recursive updates.Source: Limitations · p. 9
  • The extent of continuation reuse depends on whether matching visits are available within the rollout batch.Source: Limitations · p. 9
  • The authors identify refining matching with relevant historical information while maintaining sharing coverage, and accounting for estimate uncertainty, as future work.Source: Limitations · p. 9

Use of Generative AI Tools

The section reports the authors’ use of generative AI tools and describes the WebShop and ALFWorld evaluation settings.

Use of Generative AI Tools

  • Generative AI tools assisted with language refinement, figure preparation, and coding; the authors reviewed the content and take responsibility for the methodology, results, and conclusions.Source: Use of Generative AI Tools · p. 9

A.1 WebShop

  • WebShop experiments use a text-rich interface in a 1,000-product environment, with 500 evaluation goals, up to 15 interactions per episode, and task score and success rate as metrics.Task score measures mean match score multiplied by 100; success rate measures episodes with a perfect match score.Source: §A.1 · p. 12

A.2 ALFWorld

  • ALFWorld evaluation uses 140 held-out seen games across six household-task categories, with up to 50 interactions per episode and success defined by completing all goal conditions.Source: §A.2 · p. 12

B Implementation Details

The implementation section specifies the training and evaluation setup, while the algorithm details describe ConCal’s rollout-batch inputs and policy-update procedure.

B Implementation Details

  • ConCal is implemented in verl-agent with FSDP for policy training and vLLM for rollout generation, and Qwen2.5-1.5B-Instruct is fully fine-tuned on four H100 GPUs.The configuration uses AdamW with a constant learning rate.Source: B Implementation Details · p. 12
  • Evaluation selects the checkpoint with the highest validation success rate among the final three saved checkpoints, and main results average three independent runs.Checkpoints are saved and evaluated every five updates.Source: B Implementation Details · p. 12

C Algorithm Details

  • Algorithm 1 takes a rollout batch, episode advantages, policy references, and interaction and optimization settings, then estimates action values and updates the policy.The procedure forms task-specific visits and sharing pools, computes calibrated values and normalized step advantages, then applies PPO minibatch updates with clipping and a KL penalty.Source: C Algorithm Details · pp. 12–13

D Diagnostic Analysis Details

The diagnostics reuse frozen rollout data to evaluate relative-value calibration, evidence reuse, and computational cost. Step-advantage normalization applies a shared scale across state groups in a rollout batch, alongside reference advantages computed using GiGPO’s step-normalization rule.

D Diagnostic Analysis Details

  • Relative-value diagnostics use held-out trajectory returns to compare estimated command values within groups sharing a task, state, and remaining interaction budget.The eight trajectories are split by identifier into two sets of four; the estimator uses one set and the other supplies reference returns, then the split is swapped.Source: D Diagnostic Analysis Details · p. 13
  • Ordering statistics classify command pairs with reference gaps of at least 10^-3 as ties, agreements, or disagreements based on estimated gaps and direction.An estimated gap below 10^-3 in absolute value is classified as a tie.Source: D Diagnostic Analysis Details · p. 13

E Step-Advantage Normalization

D Diagnostic Analysis Details

  • Evidence-reuse coverage separates actions with direct cross-trajectory sharing, recursive cross-trajectory evidence, and no coverage.Direct coverage requires a sharing pool with visits from at least two trajectories and positive remaining continuation budget; percentages count each action occurrence once.Source: D Diagnostic Analysis Details · p. 14
  • Runtime comparisons replay frozen batches, measure matching and recursive calibration separately, and weight the two benchmark means equally.Calibration timing includes transferring and applying the correction on the GPU; the measurements use two warmups and seven timed repetitions per method.Source: D Diagnostic Analysis Details · p. 14

E Step-Advantage Normalization

  • Step advantages use batch-wise normalization with a scale shared across all state groups in the complete rollout batch.Reference advantages apply GiGPO’s step-normalization rule to the same calibrated values: group-wise standardization on ALFWorld and mean centering on WebShop.Source: E Step-Advantage Normalization · p. 14

F Additional Training Dynamics

Training dynamics show higher ConCal success on ALFWorld through much of training and higher average WebShop success over the final 50 updates, alongside benchmark-specific optimization patterns. The section also contrasts ConCal’s continuation calibration with related credit methods and describes the prompt templates used for agent interactions.

F Additional Training Dynamics

  • On ALFWorld, ConCal maintains higher late-training policy entropy while ending at similar reference-policy KL; on WebShop, higher late-stage success accompanies higher KL and clipping fractions.Both methods show a similar decrease followed by partial recovery in WebShop policy entropy, and their clipping fractions decline later on both benchmarks.Source: F Additional Training Dynamics · pp. 14–15
  • ConCal achieves higher ALFWorld training and validation success through much of training, and higher average WebShop training and validation success over the final 50 updates.Both methods reach nearly 100% action-format validity on both benchmarks while task-success differences persist.Source: F Additional Training Dynamics · p. 14, Figure 9 · p. 15

G.1 GraphGPO

  • For equal-cost transitions from the same state to equally distant successors, GraphGPO assigns identical distance-based step rewards, while ConCal can distinguish actions using recorded rewards and sampled continuation evidence.ConCal’s credit reflects estimated discounted returns; GraphGPO’s reflects distance to success on the merged graph.Source: §G.1 · p. 15

G.2 SALT

  • SALT averages trajectory-level advantages across matched transitions, whereas ConCal recursively calibrates continuation values before forming local credit.ConCal’s recursive propagation can reach earlier actions whose successor visits lack a direct cross-trajectory match.Source: §G.2 · pp. 15–16

G.3 GRAFT

  • GRAFT uses graph-based state-value recovery and step-advantage construction, whereas ConCal partially shares continuation evidence across visits without explicitly constructing a transition graph.The comparison also reports that Gap-matched reproduces most of ConCal’s centered-MSE reduction relative to Own-MC through magnitude adjustment, while ConCal improves ordering agreement.Source: §G.3 · p. 16

H Prompt Templates

  • Prompts include the task, current observation, available actions, and up to two recent observation–action pairs, with reasoning and a single action requested in tagged formats.WebShop uses a template without history initially and whenever the formatted prompt exceeds 13,000 characters.Source: H Prompt Templates · p. 16, Figure 11 · p. 17

Related papers

Loading 2610.37195v1…