PaperlayerView exact revision on arXiv

Source-linked AI summary

Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment

Chenyu Zhou, Qiliang Jiang, Shuning Wu, Xu Zhou

arXiv:2609.02417v1cs.LGcs.AI

TL;DR

The paper asks whether terminal-verifier credit assignment should target particular turns or cover the causal chain more broadly. It introduces verifier information density and matched shared-rollout comparisons, finding that uniform dense redistribution outperforms sparse and concentrated alternatives in the low-Vd regime.

  • Problem

    The central gap is whether per-turn credit assignment should localize terminal reward to progress turns or account for limited verifier exposure across a multi-turn causal chain.

  • Method

    The paper defines Vd = k/C and uses shared-rollout comparisons, synthetic phase-boundary experiments, matched-concentration shuffled controls, breadth sweeps, and cross-benchmark and cross-family validation.

  • Results

    Uniform dense redistribution beats sparse binary reward and concentrated progress- or random-targeted credit in the low-Vd regime, with the effect reproduced across benchmarks and model families.

  • Takeaways & Limitations

    For terminal-verifier agents in the C ≫ k regime, uniform redistribution is the zero-information coverage default that per-turn schemes must beat.

  • Takeaways & Limitations

    The τ2-bench effect is observed only within Qwen3 in this study, because it was the only open model family of that scale to clear the benchmark’s capability floor.

Abstract

from arXiv · show

Multi-turn agentic RL increasingly treats credit assignment as a targeting problem: given a terminal verifiable reward, per-turn methods localize credit onto the turns that mattered. We identify the structural quantity that predicts when this is the right move, the verifier information density V_d = k/C (the fraction of an agent's C-step causal chain whose per-turn correctness the verifier exposes), and show that terminal-state verifiers sit deep in a low-V_d regime where targeting is the wrong axis. In controlled shared-rollout comparisons on tau^2-bench that separate reward density from credit geometry, a continuous dense reward spread uniformly beats the sparse binary outcome reward (net-harmful on 4/5 seeds), while concentrating the same advantage on progress turns or on random turns is equally harmful: targeting is second-order. The mechanism is coverage: terminal-state verification collapses the observable signal to a single final-write turn (k=1 in 98% of rollouts) while success requires a 5-8 step chain of prerequisite tool calls. A synthetic phase boundary places the crossover at V_d* ~ 0.8, whereas measured V_d is ~0.15 on tau^2-bench and ~0.4 on BFCL V3; uniform also wins on BFCL, where a matched-concentration shuffled control is negative on 8/8 seeds. The effect reproduces across model families on ToolACE-2-8B (Delta = -0.048 over 32 pre-registered seeds; an independent 20-seed replication is itself significant), and a pre-registered matched-budget breadth sweep traces a monotone dose-response whose deficit vanishes only at full chain coverage, with a reward-to-go arm reaching full-coverage parity. Uniform redistribution is the zero-information coverage default that per-turn schemes must beat; we contribute the matched-concentration shuffled control that any targeting claim should clear.

1 Introduction

The paper reframes terminal-verifier credit assignment from targeting toward coverage, introducing verifier information density Vd = k/C to identify the relevant regime. In low-Vd settings, uniform redistribution covers prerequisite chains better than concentrated credit, and this pattern persists across benchmarks and model families.

  • Motivation: Terminal verification exposes only endpoint feedback for successful trajectories that depend on chains of prerequisite tool calls, creating a multi-turn credit-assignment problem.The paper contrasts endpoint scoring with single-turn RLHF, where this chain-level issue does not arise.
  • Core result: Uniform redistribution of the same terminal advantage beats heuristic concentration, while binary outcome reward is actively harmful.The comparison changes credit distribution without adding information.
  • Verifier information density: Vd = k/C measures the fraction of a causal chain whose per-turn correctness the verifier exposes; the synthetic crossover is Vd* ≈0.8, versus ≈0.15 for terminal-state verifiers.Terminal-state rollouts typically have k = 1 and C = 5–8.
  • Core result: Progress-targeted and random-targeted concentration perform similarly below uniform, indicating that concentration, rather than targeting location, is the harmful axis.The shared mechanism is insufficient coverage of the 5–8-step causal chain when observable credit collapses to k = 1.
  • Methodological contribution: The paper contributes a matched-concentration shuffled control for distinguishing targeting benefits from changes in coverage breadth and optimization dynamics.It recommends this control for per-turn credit-assignment claims.
  • Validation: The effect reproduces on BFCL V3 at Vd ≈0.4, across model families, and in a matched-budget breadth sweep where performance rises with chain coverage.A reward-to-go arm reaches full-coverage parity, while uniform redistribution is proposed as the zero-information default.

2 Setup

The setup uses shared base-policy rollouts and matched optimization to separate reward density from credit geometry. Four reward arms are compared on held-out support and official success, with paired within-run inference.

  • Environment: The experiments use τ2-bench retail database-state tasks, Qwen3 agents, GRPO with LoRA, BFCL V3, and a telecom contrast.The primary base agent is Qwen3-14B, with 8B and 4B included for scale analysis.
  • Shared-rollout isolation: Each seed supplies one shared rollout batch, from which every arm trains with the same optimizer, separating reward density from credit geometry.The geometry comparison holds data, exploration, and terminal advantage fixed while varying its turn-wise distribution.
  • Credit geometries: Binary uses sparse 0/1 outcomes; uniform assigns continuous DB-field progress to every turn; per-turn concentrates it on progress turns; shuffled moves the same concentration randomly.Per-turn and shuffled therefore test progress targeting versus random targeting at matched concentration.
  • Optimization control: A global gradient-norm clip of 1.0 equalizes applied update magnitude, so geometries differ in update direction rather than size.The per-turn arm also receives more verifier information than uniform because it observes where progress occurred.
  • Metrics and inference: Evaluation uses held-out environment-assertion support as the primary continuous metric and official task success as the co-primary binary metric.Confirmatory comparisons are paired within runs over 95 task–seed pairs.
  • Figure 2: Figure 2 displays paired per-seed contrasts for retail-14B, with faint markers for seed differences and diamonds for means.The figure reports binary degradation, uniform improvement, harmful concentration, and indistinguishable ternary performance.

3 Density is necessary but not sufficient

Dense reward is necessary, but concentration remains harmful when the verifier exposes only a small fraction of a longer causal chain. Across controlled comparisons and synthetic sweeps, uniform coverage wins until verifier information density approaches most of the chain.

  • Reward density: +0.079: Uniform dense exceeds binary reward on gradable assert-support, while binary degrades the base policy and uniform improves it.The uniform–binary gap is significant across 95 task–seed pairs, and official success differs by +5.2pp.
  • Reward density: 96%: Ternary dense reward retains nearly all of uniform dense’s advantage over binary, showing that non-degenerate feedback matters more than fine resolution.The ternary scheme uses levels 0, 0.5, and 1.
  • Credit geometry: −0.053: Progress-targeted concentration underperforms uniform on assert-support, and the deficit persists across the trainable dose range and on BFCL V3.The BFCL V3 matched comparison reports −0.036; the same total credit budget separates coverage from scale.
  • Credit geometry: +0.014: Random-targeted credit lands within noise of progress-targeted credit, while both remain below uniform, indicating concentration rather than target choice drives the deficit.The shuffled comparison is a null on targeting, not certified equivalence; its interpretation follows the shared coverage mechanism.
  • Coverage mechanism: k = 1 versus C = 5–8: Real rollouts concentrate observable progress on the terminal write, although success requires prerequisite reads and other causal tool calls.In 98% of rollouts the progress signal lands on one turn; static data-flow checks show precursor reads are physically necessary for the final write.
  • Verifier information density: Vd ≈0.81: Targeting’s synthetic crossover occurs only when the verifier exposes roughly four-fifths of the causal chain; real terminal-verifier agents operate near Vd ≈0.15.Across chain lengths C = 4–10, the crossover remains in [0.75, 0.91], and uniform stays flat while targeted performance rises with coverage.
  • Verifier information density: Vd ≈0.4: BFCL V3 has the highest measured per-turn checker density in the benchmark census, yet no widely used benchmark naturally exposes high Vd.The terminal-verifier majority sits near Vd ≈0.1–0.2, making uniform redistribution the zero-information default in this setting.
  • Prescription: Uniform redistribution is the default coverage strategy: fixed-budget concentration cannot beat it when C ≫ k, whereas targeting can become useful once the verifier supplies most of the chain’s per-turn coverage.The paper frames learned calibration or oracle process rewards as complementary ways to supply missing per-step information.

6 Validation and boundary conditions

Validation across capability, model family, benchmark, and coverage breadth supports a common low-verifier-density regime in which uniform credit outperforms concentration. The effect disappears below a capability or signal-resolution floor and reaches parity only with full chain coverage.

  • Capability floor: A capability floor limits the effect: 4B completes prerequisite chains in 24% of rollouts versus approximately 70% for 14B, leaving too few completed chains for credit geometry to matter.The task-required chain is unchanged; the difference is completion frequency.
  • Scale reproduction: Above the floor, dense rewards beat binary by +0.048 at 8B and +0.079 at 14B, while per-turn minus uniform is −0.050 and −0.053, respectively.Three scale anchors—4B, 8B, and 14B—argue against a 14B-specific artifact.
  • Second-benchmark reproduction: On BFCL V3, per-turn minus uniform is −0.036 and shuffled minus uniform is −0.034; shuffled minus per-turn is +0.002, reproducing the targeting null.All arms share each seed’s training rollouts and one fresh evaluation batch.
  • Cross-family reproduction: ToolACE-2-8B reproduces concentration below uniform across 32 pre-registered seeds, with a pooled delta of −0.048 and 29/32 negative seeds.The independent 20-seed replication is significant on its own: delta −0.054, t = −5.73.
  • Breadth sweep: At fixed total credit, concentrating on one turn costs −0.048, widening to checker-visible steps recovers +0.031, and full chain coverage reaches parity at +0.010.Shuffling checker-visible credit changes performance by a statistically indistinguishable +0.014 while both concentrated arms remain below uniform.
  • Verifier-density regime: Terminal-state verification commonly collapses observable progress to k=1; measured V_d is approximately 0.15 on τ 2-bench and approximately 0.4 on BFCL V3, below the crossover near 0.8.The coverage structure k=1, C≫k also holds across retail-14B, retail-4B, and telecom-14B.

7 Related work

The paper distinguishes its coverage account from prior work that adds or reshapes intermediate supervision. It argues that per-turn methods help insofar as they restore causal-chain coverage, not merely because they localize credit.

  • Intermediate supervision: Prior work adds turn-level rewards, critics, calibration, process supervision, or hindsight relabeling under the premise that better localization is missing.The paper frames its contribution as isolating concentration from targeting rather than denying that richer feedback can help.
  • Coverage interpretation: The proposed distinction is that learned per-turn signals can help by raising effective, data-driven V_d when they recover coverage lost under structural k=1 verification.Such gains remain compatible with the paper’s high-V_d prediction, subject to model bias and localization risk.
  • Benchmark verifier structures: Across agentic benchmarks, terminal-state verification structurally implies k=1 and V_d = 1/C ≤ write-fraction when the verifier runs once on the terminal state.The cited harnesses include SWE-bench, WebArena state-scored tasks, and OSWorld.
  • Scope relative to adjacent work: The paper separates its multi-turn rollout coverage account from related work on reward richness, token-level uniformity, and module-level credit routing.At C≫k, the cited module-level result likewise links losses to starving bottleneck modules and removes harm with a coverage floor.

8 Limitations

The evidence is strongest for terminal-verifier agents above the capability floor, with real-model comparisons focused on first policy updates. Some supporting batteries are directional or limited to particular benchmark and model-family settings.

  • Scope: The credit-geometry effect is measured within above-floor Qwen3 on τ 2-bench, while cross-family reproduction is reported on BFCL V3 with ToolACE-2-8B.The τ 2-bench result was observed only within Qwen3 in the tested scale class.
  • Training horizon: Longer-horizon training dynamics are outside scope because the real-model comparisons isolate the first policy update.This bounds direct conclusions about multi-update training behavior.
  • Evidence strength: The Qwen3-8B battery is directional because its arms were trained in separate runs.The matched-budget sweep separates coverage from update scale on the real model.
  • Methodological boundary: Targeting claims should use a matched-concentration shuffled control, because changing credit geometry jointly changes concentration and optimization dynamics.Without that control, apparent per-turn gains remain confounded between localization, coverage, and effective update magnitude.

A The per-turn deficit holds across the trainable dose range

Per-turn concentration underperforms uniform across the trainable update-strength range, including substantially higher-dose settings. The deficit therefore is not confined to the main optimizer dose, although very high doses can destabilize training.

  • Dose sweep: Across the full trainable dose range, per-turn concentration remains worse than uniform on held-out assert-support.Table 4 defines negative delta as per-turn worse.
  • Main dose: At the gentle main dose, per-turn trails uniform by 0.053 across 5 seeds.This is the baseline comparison for the dose-strength analysis.
  • High-dose behavior: At 40× effective dose, both arms degrade and the per-turn deficit widens to 0.20 on the seed that does not fully collapse.Under two-epoch updates, raising the learning rate instead diverges because off-policy ratios blow up.

B Between-batch evaluation variability, and why all comparisons are within-run paired

The evaluation protocol quantifies variability from repeating the same uniform-adapter evaluations and uses paired, hierarchical inference for within-run comparisons.

  • Four uniform adapters were re-evaluated on the same 19-task set under an identical configuration to measure estimator variability.The repeated batch used the same split, sampling, serving stack, and user simulator.
  • +0.0789 mean difference, with percentile 95% CI [+0.0085, +0.1492], was obtained for uniform − binary using a paired hierarchical bootstrap.The bootstrap resampled five seeds first, then 19 paired tasks within each selected seed, over 20,000 replicates.
  • A matched-batch paired design is recommended for per-turn credit-assignment comparisons at small evaluation budgets.Auxiliary arms from separate runs are reported directionally rather than as effect-fraction estimates.

C Synthetic environment specification

The synthetic environment is a controlled episodic task with known causal structure, while training distributes group-normalized terminal advantages across turns using configurable credit geometries.

  • Each episode has T=12 turns, with horizon variants T=24 and T=48, fixed random unit-norm features ϕt ∈R8, and K=5 actions.The V_d sweep uses K=10 to keep uniform training below its performance ceiling.
  • A group of G=24 rollouts receives group-normalized terminal advantages Ai, which are distributed across turns before a policy-gradient update.The training procedure uses GRPO-style group normalization.
  • Uniform spreads Ai/T across all turns, whereas concentrated credit assigns the same budget across k selected turns.The concentrated arm uses k=3 in the C-sweep.

D Real-model training configuration

Real-model comparisons use a common GRPO/LoRA stack, while dense reward arms produce more trainable group-level signals than binary rewards.

  • All real-model comparisons use the same GRPO/LoRA stack, with arms differing only in the reward argument within each benchmark.Experiments span τ 2, BFCL, and ToolACE settings.
  • The configuration is documented in the real-model training and evaluation table.

E Metric definition and evaluation protocol

The evaluation averages rollout-level assert-support over a frozen gradable task subset, alongside official task success and causal-chain diagnostics from retained traces.

  • Assert-support is averaged over four rollouts per task and then macro-averaged across 19 frozen gradable tasks.Reported contrasts average within-seed differences of task-macro means; rollout or task–seed cells are not treated as independent seeds.
  • The frozen subset contains 19 held-out tasks selected by the pre-results rule |Gq| ≥2, while official success uses all 20 held-out tasks.
  • Causal-chain measurements are robust to three definitions of C, with all plug-in V_d values at most 0.167.The definitions count every tool call, distinct tools, or data-flow ancestors of the terminal write.
  • 94.7% of unique identifiers consumed by terminal writes do not appear in the user instruction.This indicates that terminal-write arguments generally require information beyond the instruction alone.
  • On official task success, per-turn − uniform equals −0.063, with 4/5 seeds negative and one tie.The result reproduces the geometry deficit on the deployed metric.

G The pre-registered credit-breadth sweep on ToolACE

A pre-registered breadth sweep matched terminal-advantage budgets across increasingly broad turn sets, using shared rollout batches and common evaluations. Breadth improved monotonically on ToolACE, while reward-to-go reached parity with full coverage on BFCL.

  • Experimental design: The sweep compared single-turn, shuffled, executed-tool-call, all-turn, and doubled-learning-rate checker-visible arms under identical training and evaluation configurations.Archived uniform and checker-visible-steps adapters were also re-evaluated in one common held-out batch per seed.
  • Experimental design: Every budget-matched arm redistributed the same group-normalized terminal advantage, enforcing identical total absolute weight per rollout.This isolates credit breadth from the amount of learning signal assigned.
  • Experimental design: The BFCL fallback used all executed tool calls as the widest concentrated arm because a data-flow oracle was not constructible with median provenance set size 1.The companion experiment evaluated breadth arms and reward-to-go across five seeds using frozen Qwen3-8B rollout batches.
  • ToolACE results: +0.031 and +0.027 were the adjacent-step ToolACE contrasts from spike to checker-visible and checker-visible to all tool calls, respectively.The 2×-learning-rate arm had post-update KL 0.60 versus uniform’s 0.006, bracketing the deficit from above; archived contrasts reproduced −0.050 versus −0.054.
  • BFCL results: 95% CI [−0.019, +0.069] was the normalized all-turns BFCL result, reaching parity with full coverage and scoring +0.045 over checker-visible concentration.It also scored +0.020 over progress-targeted concentration, while reward-to-go scored highest among eight arms.
  • BFCL results: The concentrated-arm BFCL deficits were directionally consistent, with the spike at −0.018 and negative on 4/5 seeds, while the 20-seed ToolACE sweep resolved the intermediate breadth rungs.Reward-to-go propagates the terminal signal to every turn preceding the last progress point.
Loading 2609.02417v1…