PaperlayerView exact revision on arXiv

Source-linked AI summary

DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter

Xuyan Ye, Xiyan Fu, Shengda Fan, Jiawei Cui, Yankai Lin, Wei Lu

arXiv:2610.17666v1cs.LGcs.AIcs.CL

TL;DR

Sparse terminal rewards make step-level credit difficult, and existing approaches can rely on state matching or costly step scoring. DeltaTTG learns a shared progress meter from recorded trajectories and uses changes in its frozen estimates to redistribute outcome-derived advantages. It achieves the highest average score with both evaluated backbones and supports test-time action selection and cross-domain meter transfer.

  • Problem

    Sparse, delayed terminal rewards provide little information about which steps helped or hurt, while within-group comparisons depend on comparable states and diverse outcomes.

    Source: §1 · pp. 1–2

  • Method

    DeltaTTG trains a reusable Token-to-Go progress meter on diverse recorded trajectories and uses changes in its frozen remaining-work estimates to redistribute outcome-derived advantages across steps.

    Source: §1 · p. 2

  • Results

    DeltaTTG achieves the highest average score with both backbones: 72.86 vs. 70.41 for the strongest baseline with Qwen3-4B, and 66.49 vs. 61.69 with Llama-3.1-8B-Instruct.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    The reusable meter supports test-time action selection and cross-domain transfer while keeping actor-training cost close to GRPO.

    Source: §6 · p. 8

    Evaluation covers only two actor model families and several domains, so broader transfer requires further validation and may need adaptation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps. Within-group step comparisons depend on reliable state matching and sufficiently diverse sampled outcomes, while direct step scoring may require costly external evaluators, human annotations, or task-specific heuristics. We introduce DeltaTTG to learn a reusable progress signal through a shared progress meter, using low-cost supervision from diverse policies' recorded trajectories without task-specific progress rules. Its supervision target, Token-to-Go (TTG), the number of tokens remaining to be generated until task completion, serves as an observable proxy for remaining completion effort. During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale. On ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG improves the average score over GRPO by 5.7 and 7.2 points with Qwen3-4B and Llama-3.1-8B-Instruct, respectively, outperforms all compared RL baselines on average, and keeps actor-training GPU-hours close to GRPO. Further experiments show improved sample efficiency and out-of-domain performance, and support the meter's ability to distinguish step quality and guide test-time action selection across actor sizes. The code is available at https://github.com/Xuyan923r/DeltaTTG.

1 Introduction

Sparse terminal rewards make it difficult to assign credit to individual steps, while existing step-scoring approaches have constraints. DeltaTTG learns progress from recorded trajectories and uses it to redistribute outcome-derived credit across steps.

1 Introduction

  • Terminal rewards arrive after long decision sequences and provide little information about which steps helped or hurt.This makes step-level credit assignment difficult.Source: §1 · p. 1
  • Within-rollout comparisons depend on comparable states and diverse outcomes, while direct step scoring may require costly evaluators, annotations, rollouts, or task-specific heuristics.Similarity-based grouping can pool states with different decision-relevant conditions.Source: §1 · pp. 1–2
  • Token-to-Go (TTG), the number of tokens remaining until success, provides an observable proxy for completion effort from recorded trajectories across policies.The paper hypothesizes that beneficial actions leave fewer tokens to completion on average than harmful actions from the same state.Source: §1 · p. 2
  • DeltaTTG trains a reusable progress meter from recorded trajectories and uses changes in its frozen estimates to distribute trajectory outcome advantages across steps.The evaluation spans ALFWorld, PaperSearchQA, and BFCLv3 across two model families and parameter scales.Source: §1 · p. 2

2 Related Work

Prior process-supervision approaches either directly score steps or redistribute outcome-based credit. DeltaTTG learns remaining-work estimates from cross-policy trajectories and applies their changes to step credit.

2 Related Work

  • Process supervision includes direct step scoring with evaluators, annotations, verifier feedback, search or reference reasoning, task-specific heuristics, and on-policy distillation.A second category redistributes outcome-based credit across process units.Source: §2 · p. 2
  • Some outcome-credit methods compare sampled actions under shared states, histories, or transition dynamics, while others learn step contributions that reconstruct final outcomes.DeltaTTG instead uses cross-policy Token-to-Go labels and does not require matched peer states.Source: §2 · p. 2
  • The framework first learns a shared TTG progress meter from recorded trajectories, then uses its frozen estimates for actor credit assignment.The overview describes the meter as trained through SFT and RL with continuation outcomes and remaining-token labels.Source: Figure 2 · p. 3

3 Methodology

DeltaTTG trains a shared Token-to-Go meter on recorded trajectories and uses its frozen remaining-work estimates to redistribute outcome-derived actor advantages across steps.

3.1 Training a Token-to-Go Progress Meter

  • Recorded trajectories from diverse policies are split at assistant-action boundaries into prefixes and suffixes, supplying completion outcomes and successful-trajectory TTG labels.Failure labels represent recorded outcomes, not intrinsic unsolvability; pooled observations train a shared remaining-work estimate.Source: §3.1 · p. 3
  • The meter predicts completion and nonnegative remaining work from histories, with SFT teaching structured outputs and an RL refinement stage rewarding correct completion forecasts and remaining-work estimates.The trained meter is frozen during actor optimization.Source: §3.1 · pp. 3–4

3.2 DeltaTTG Credit Assignment

  • For actor credit assignment, the frozen meter estimates state costs, and each action’s signal is the reduction in predicted remaining work; positive changes indicate progress and negative changes regression.A fixed cost represents unsuccessful-completion estimates, distinguishing failure from low-effort completion.Source: §3.2 · p. 4
  • Progress or regression weights are allocated according to the trajectory advantage’s sign, then mixed with a uniform step share to reduce sensitivity to estimation errors.The method broadcasts step weights to trainable tokens and normalizes token-level multipliers.Source: §3.2 · p. 4
  • Trajectory advantages set the update direction, while step weights set its strength; the cap limits amplification from large C-induced progress changes.The reported settings are ρ = 0.2 and M = 5; the clipped actor loss changes only advantage allocation relative to GRPO.Source: §3.2 · p. 5

4 Experiments

The experiments compare DeltaTTG with RL baselines across three task benchmarks, two actor backbones, and ablations of its components. DeltaTTG leads on average, improves sample efficiency, and shows competitive training cost and out-of-domain performance.

4.1 Experimental Setup

  • The evaluation uses Qwen3-4B and Llama-3.1-8B-Instruct across ALFWorld, PaperSearchQA, and BFCLv3, with tool-use training on AgentWorldModel.The benchmarks cover interactive decision making, scientific-paper search and reasoning, and function calling and multi-turn tool use.Source: §4.1 · p. 5

4.2 Main Results

  • DeltaTTG achieves the highest average score with both backbones: 72.86 vs. 70.41 for Qwen3-4B and 66.49 vs. 61.69 for Llama-3.1-8B-Instruct.It has the highest mean, including ties, on 13 of 16 open-backbone metrics; its largest reported gains are on BFCLv3 tool use, while it trails the strongest baseline on Hallucination/Irrelevance with both backbones.Source: §4.2 · p. 5, Table 1 · p. 6
  • DeltaTTG reaches approximately 80% test success by step 75, whereas GiGPO requires around 125 steps, or roughly 40% fewer optimization steps in this training run.The response-token comparison excludes prompt and progress-meter tokens, so it measures actor response-token efficiency rather than total training cost.Source: §4.2 · p. 5, Figure 3 · p. 6

4.3 Ablation Studies

  • The full DeltaTTG achieves higher average success than all six ablations on both ALFWorld splits.The ablations favor TTG remaining-work supervision over outcome-value supervision and support completion predictions plus token estimates, sign-aware reversal, meter training, and uniform mixing.Source: §4.3 · p. 6, Table 2 · p. 7

4.4 Computational Cost

  • On ALFWorld Seen with Qwen3-4B, DAPO and DeltaTTG tie for the highest success rate, while DeltaTTG achieves it at roughly GRPO's GPU-hour cost.DAPO uses 2.7× GRPO's GPU-hours; DeltaTTG's meter requires 10.45 GPU-hours of RL training and is reused across actor models.Source: §4.4 · pp. 6–7, Figure 4 · p. 7

4.5 Out-of-Domain Policy Generalization

  • DeltaTTG achieves the highest performance among compared methods on BrowseComp-Plus, τ 2-bench, and Terminal-Bench 2.0.The reported results suggest task-progress-based credit assignment improves in-domain performance while preserving capabilities on these evaluated out-of-domain benchmarks.Source: §4.5 · p. 7

5 Analyzing the Progress Meter

The section evaluates whether Token-to-Go supervision captures step quality, whether a fixed progress meter guides test-time action selection, and whether its signals transfer across domains.

5.1 Token-to-Go as Progress Supervision

  • 56.91% accuracy with Token-to-Go supervision exceeds 50.83% from direct prompting on held-out step-quality annotations.With neutral steps excluded, random guessing has an expected accuracy of 50%; the jointly trained meter uses four datasets, unlike the domain-specific meters used for actor training.Source: §5.1 · p. 7
  • Token-to-Go achieves the highest step-pooled Overall accuracy on both validation and test splits among the compared supervision targets.The comparisons include Future Score and Step-to-Go; the ALFWorld ablations also report lower Unseen success when using completion logits or binary feasibility instead of Token-to-Go.Source: §5.1 · p. 7, Table 3 · p. 8

5.2 Test-Time Guidance across Actor Scales

  • Across Qwen3-1.7B, 4B, 8B, and 14B actors, progress-guided rollouts have higher success rates and fewer average interaction steps than natural or random-selection rollouts.Each step considers four candidate actions; the random-selection comparison indicates gains are not due to the candidate budget alone.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 Cross-Domain Progress-Meter Transfer

  • An ALFWorld-trained meter used without target-domain adaptation outperforms GRPO on PaperSearchQA Original, Paraphrased, and Overall evaluation.Figure 5d assesses transfer of the meter to PaperSearchQA optimization.Source: §5.3 · p. 8, Figure 5 · p. 8

6 Conclusion

DeltaTTG uses a reusable Token-to-Go meter trained on recorded trajectories to redistribute outcome-derived advantages, improving over strong RL baselines at actor-training cost close to GRPO while supporting test-time action selection and cross-domain transfer.

6 Conclusion

  • DeltaTTG redistributes outcome-derived advantages with a reusable Token-to-Go meter trained on recorded trajectories, improving over strong RL baselines at actor-training cost close to GRPO.The approach also supports test-time action selection and cross-domain meter transfer.Source: §6 · p. 8

Limitations

The study identifies limits in evaluation breadth, sensitivity analysis, and measurement of training variability.

Limitations

  • Evaluation covers two actor model families and several domains; broader transfer remains unvalidated and may require adaptation.Source: Limitations · p. 9
  • The study does not establish robustness to C, despite an audit finding that C does not affect credit allocation for most successful trajectories.Task-specific values across three training domains yield higher aggregate scores than the compared RL baselines.Source: Limitations · p. 9
  • Reported standard deviations come from repeated evaluations of one trained checkpoint, leaving variability across independent training runs unquantified.Source: Limitations · p. 9

A Benchmark

The benchmark suite covers household interaction, biomedical evidence retrieval, and tool use, with additional out-of-domain tests for information seeking, terminal tasks, and service interactions.

ALFWorld

  • ALFWorld uses 3,553 training tasks and evaluates on Seen (140 tasks) and Unseen (134 tasks) splits across six household task types.Source: A Benchmark · p. 11

PaperSearchQA

  • PaperSearchQA tests biomedical question answering over 16 million PubMed abstracts, using 54,907 training examples and 5,000 test examples.The test set contains 2,500 Original and 2,500 Paraphrased questions, with exact-match success reported separately.Source: A Benchmark · p. 11

AgentWorldModel and BFCLv3

  • AgentWorldModel supplies 9,850 actor-training tasks across 985 environments, while BFCLv3 tests function selection, argument generation, and multi-turn tool use.AgentWorldModel is not used as a test benchmark; BFCLv3 includes Non-Live, Live, Multi-Turn, and Hallucination/Irrelevance categories.Source: A Benchmark · p. 11

Out-of-domain benchmarks

  • Out-of-domain tests include 830 BrowseComp-Plus questions, 278 τ 2-bench service tasks, and 89 Terminal-Bench 2.0 tasks.They assess answer accuracy, overall and domain-level success rates, and pass@8, respectively.Source: A Benchmark · pp. 11–12

B Baseline Methods

The appendix describes prompt-based and RL baselines, the shared training infrastructure, computational resources, and repository-default hyperparameters.

B Baseline Methods

  • Prompt-based baselines assess pretrained models without updates, while RL baselines optimize policies from environment feedback; ReAct interleaves reasoning and actions, and Reflexion retries failures with reflections.Reflexion allows up to 10 attempts on ALFWorld and PaperSearchQA and up to three on BFCLv3.Source: B Baseline Methods · p. 12
  • GRPO uses trajectory-level group-relative advantages, DAPO adds asymmetric clipping and dynamic sampling, and GiGPO and HGPO derive step-level advantages through state or history matching.The listed settings include state-similarity matching for GiGPO and progressively longer history groups for HGPO.Source: B Baseline Methods · p. 12

C.1 Training Infrastructure

  • All agentic RL methods use verl-agent3 infrastructure, providing a common rollout and distributed-training basis for comparing methods.DeltaTTG is integrated into the same training pipeline to keep rollout and optimization settings consistent.Source: §C.1 · p. 12

C.2 Computational Resources

  • AgentProcessBench analysis runs on AMD Instinct MI325X GPUs; all other experiments use NVIDIA H200 GPUs.Source: §C.2 · p. 12

C.3 Training Hyperparameters

  • Repository defaults cover common and method-specific training settings; ALFWorld and PaperSearchQA meters use one SFT epoch followed by GRPO refinement and are shared across actor backbones.The stated learning rates are 10^-5 for SFT and 3 × 10^-6 for GRPO refinement, with group size 8.Source: §C.3 · p. 13

C.4 Evaluation Metrics

The evaluation section defines task-specific metrics and repeated-run reporting, then specifies the actor and serving context for out-of-domain tests.

C.4 Evaluation Metrics

  • ALFWorld is scored by success rate, PaperSearchQA by exact match against accepted answers, and BFCLv3 by its official category scores.For repeated Table 1 evaluations, reported values are means and population standard deviations over three test runs; entries without standard deviations are point estimates.Source: §C.4 · p. 13

C.5 Out-of-Domain Evaluation Settings

  • Out-of-domain evaluations use Qwen3-4B actors, with BrowseComp-Plus checkpoints trained on PaperSearchQA and Terminal-Bench 2.0 and τ 2-bench checkpoints trained on AgentWorldModel.The settings also specify thinking mode, a GPT-5.2 low-reasoning-effort user simulator for τ 2-bench, and non-solo mode across its three domains.Source: §C.5 · p. 13

D.1 Progress-Meter Training Data and Reuse

These appendix sections describe progress-meter data and training, advantage redistribution and actor optimization, detailed evaluations and ablations, and test-time action guidance.

D.1 Progress-Meter Training Data and Reuse

  • Progress meters are trained per task category from recorded trajectories and prefixes collected across source models, then frozen and shared across Qwen3-4B and Llama-3.1-8B-Instruct actors.Some dataset reference answers are also incorporated into the training data.Source: §D.1 · p. 13

D.2 Token-to-Go Progress Meter Training

  • Meter SFT pairs observed prefixes with completion and remaining-token labels; GRPO refinement trains the meter, whose rewards are never added to actor rewards.The meter remains frozen throughout actor optimization.Source: §D.2 · p. 13

D.3 Group-Relative Advantage Normalization

  • Step-expanded group statistics weight trajectories by their action counts, and bounded token-credit normalization gives trainable tokens unit-mean weights capped at M.The original group-relative advantage is initially assigned uniformly to each trainable token before progress-based redistribution.Source: §D.3 · p. 13, §D.4 · p. 14

D.5 Actor Optimization Objective

  • Actor optimization applies a clipped token-level policy objective to redistributed advantages, with dual clipping for negative advantages and configured KL and entropy terms.The reported clipping interval is [0.8, 1.28], and the loss uses token-mean reduction.Source: §D.5 · pp. 14–15

D.6 Algorithm

  • After rollout and group-relative advantage calculation, DeltaTTG batch-scores decision states with a frozen meter, redistributes advantages across actions and tokens, and updates the actor.The baseline comparison keeps the policy loss and KL configuration fixed.Source: §D.6 · p. 15, §E.2 · p. 15

E.3 Detailed Training Curves

  • The detailed appendices provide ALFWorld Seen/Unseen category breakdowns, τ 2-bench domain success rates, and training curves plotted against both steps and cumulative response tokens.Figure 7 has 12 benchmark–backbone panels; its AgentWorldModel rows show training-batch success rates, and evaluation overlays summarize three seeds where shown.Source: §E.1 · p. 15, §E.2 · p. 15, §E.3 · p. 15

E.4 Detailed Ablation Settings

  • Ablations compare the full method with removing meter training or uniform mixing, and with replacing TTG supervision by trajectory outcomes to isolate the target’s contribution.The outcome-only variant uses the same trajectories and training configuration but substitutes each trajectory’s final score for its prefixes’ TTG labels.Source: §E.4 · pp. 17–18
  • Other ablations substitute completion logits or binary feasibility for TTG, or remove sign-aware reversal; feasibility-based raw progress changes only when adjacent binary predictions differ.Source: §E.4 · pp. 17–18

E.6 AgentProcessBench Progress-Meter Analysis

The analysis trains meters on AgentProcessBench trajectory prefixes and evaluates whether predicted progress signals align with annotated step quality; a separate case study presents two AgentWorldModel traces.

E.6 AgentProcessBench Progress-Meter Analysis

  • AgentProcessBench uses 1,000 trajectories across BFCL, GAIA, HotpotQA, and τ 2-bench, split by dataset and outcome into train, validation, and test sets.Prefixes remain within their trajectory’s split, and labels are derived from recorded trajectories without additional rollouts.Source: §E.6 · p. 18
  • Separately fine-tuned meters predict Future Score, Step-to-Go, or Token-to-Go, and their step-quality judgments are evaluated against annotations on seen tasks.Neutral steps are excluded; overall accuracy pools evaluated steps, with random guessing at 50% and direct prompting at 50.83% on the held-out test split.Source: §E.6 · p. 18

F Case Study

Loading 2610.17666v1…