VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents

1 INTRODUCTION

• Existing approaches either assign one trajectory-level advantage across turns or estimate token-level values from delayed feedback, while turn-level methods leave within-response generation decisions undifferentiated.These gaps motivate coordinating credit at both response and token scales.Source: §1 · p. 2
• The mixed configuration achieves higher mean success rates than PPO* and Turn-PPO on both ALFWorld and WebShop, suggesting complementary information from the two credit granularities.Figure 1(a) compares the alternatives; its bars report means and standard deviations over three decoding seeds per checkpoint.Source: §1 · p. 2, Figure 1 · p. 2
• VETTA learns turn- and token-level values with separate heads on a shared lightweight critic, then combines turn advantages with within-response-centered token residuals for token-wise PPO updates.The critic retains early Transformer blocks from the actor’s pretrained checkpoint to reduce value-learning computation.Source: §1 · pp. 2–3
• At 7B, VETTA reaches 95.5% success on ALFWorld and 76.0% on WebShop, with the highest overall success rates among critic-based and critic-free baselines across both benchmarks and model sizes.At 1.5B, it reaches 91.9% on ALFWorld and 73.8% on WebShop.Source: §1 · p. 2
• The two-block critic achieves higher mean success rates than the 28-block critic on both benchmarks while reducing median critic-side time by approximately 87% on ALFWorld and 91% on WebShop.The comparison examines mixed-credit critics with 2, 8, or 28 Transformer blocks.Source: §1 · pp. 2–3, Figure 1 · p. 2

2.1 MULTI-TURN LLM AGENT INTERACTION

• A multi-turn interaction can be represented as a sequence of complete responses or as a finer sequence of generated-token decisions.The agent receives context, generates a response, receives a scalar reward and a new observation, and incorporates that observation into the next context.Source: §2.1 · p. 3, §2 · p. 3

2.2 PPO AND GENERALIZED ADVANTAGE ESTIMATION

• The same interaction supports turn-level decisions corresponding to complete responses and token-level decisions corresponding to individual generated tokens.Source: §2.2 · p. 3
• PPO uses advantages to guide policy updates; generalized advantage estimation uses rewards and value predictions along a sampled trajectory to estimate them.The advantage is action value minus state value, and PPO’s probability-ratio clipping limits incentives for large policy changes.Source: §2.2 · p. 3

3 METHOD

• VETTA uses a shared critic to estimate values for actor-collected multi-turn trajectories, then uses the resulting advantages for critic regression and PPO policy updates.Turn- and token-level value heads share the critic backbone and are trained with their respective return targets.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 DUAL-GRANULARITY VALUE ESTIMATION WITH A LIGHTWEIGHT CRITIC

• Separate heads estimate value before each complete response and before each generated token, using representations from a shared causal-attention backbone.The critic retains the token embedding, its first d Transformer blocks, and final RMSNorm; retained components remain trainable.Source: §3.1 · p. 4

3.2 ADVANTAGE ESTIMATION

• VETTA computes GAE along turn and token sequences, then combines turn-level credit with a within-response-centered token residual.Token-level recursion follows generated tokens within and across responses, and the implicit successor value and advantage are zero at rollout end.Source: §3.2 · p. 5

3.3 CRITIC AND ACTOR UPDATES

• The critic regresses on separate turn- and token-level return targets, while the actor retains token-wise PPO ratios and clipping and uses the composed advantage.Advantages and targets are computed before optimization and held fixed during updates.Source: §3.3 · p. 5

4.1 EXPERIMENT SETUP

• The experiments cover six-category household tasks in ALFWorld and search and navigation on a simulated shopping website in WebShop, comparing prompting and RL baselines.The baselines include non-RL prompting, critic-free RL methods, token-level PPO variants, and turn-level Turn-PPO.Source: §4.1 · p. 6

4.2 EXPERIMENTAL RESULTS (RQ1)

• At 1.5B, VETTA exceeds the strongest alternative by 5.8 percentage points on ALFWorld and 6.4 on WebShop; at 7B, it reaches 95.5% and 76.0% success, respectively.At 7B, it retains the highest mean overall success rate among critic-based and critic-free baselines on both benchmarks.Source: §4.2 · p. 7

4.3 FURTHER ANALYSIS

• Residual composition C4 achieves 72.8% WebShop success, above direct addition C3 at 55.0%, supporting within-response centering and combined credit for exact completion.The comparison uses matched dual-head runs; single-branch controls also favor combining credit levels for exact task completion.Source: §4.3 · pp. 7–8
• On ALFWorld and WebShop, the shared two-head critic leads the tested critic structures, with success rates of 93.6% and 72.8%, respectively.Independent critics reach 89.3% and 58.4%, while a shared one-head critic reaches 91.4% and 65.0%; WebShop task scores follow the same ordering. These single-run comparisons do not isolate representation sharing alone.Source: Figure 3 · p. 8, §4.3 · p. 8
• With two critic blocks, median value-inference-plus-update time is 19.7 versus 152.0 seconds on ALFWorld and 7.9 versus 92.2 seconds on WebShop, compared with 28 blocks.The two-block critic also has the highest mean success on both benchmarks; eight-block runs lie between the two- and 28-block timing endpoints, and more layers do not necessarily improve success.Source: Figure 4 · p. 8, §4.3 · pp. 8–9

Turn-Level Credit Assignment

• Turn-level methods assign credit to complete actions using sampled returns, transition graphs, or learned values, while distinguishing contributions across actions.Examples include Turn-PPO, which applies PPO to complete responses, and ArCHer, which learns utterance-level values to guide a token policy.Source: §5 · p. 9

Token-Level and Joint Credit Assignment

• Token-level and joint-credit methods include cross-response token GAE, executable-token weighting, teacher-feedback adjustments, and combined turn/token advantages.HyGAE combines turn- and token-level advantages with a unified value function and mixed return targets.Source: §5 · p. 9
• VETTA keeps separate supervision for turn and token value heads, adding a centered token residual so the turn advantage sets the composed signal’s within-response mean.The token residual averages to zero within each response and adjusts credit among its generation decisions.Source: §5 · p. 9

Critic Architecture and Efficiency

• Critic designs vary in representation sharing and computation; VETTA uses early Transformer blocks from the actor’s pretrained checkpoint as its shared critic backbone.Related approaches include smaller-critic ensembles and lightweight value probes on actor representations.Source: §5 · p. 9

• VETTA estimates turn- and token-level values with separate heads on a shared lightweight backbone, combining turn advantages with centered token residuals for policy optimization.The conclusion identifies credit granularity and critic efficiency as complementary design choices and proposes task-tailored value models as future work.Source: §6 · p. 9

• Generative AI tools assisted with drafting, literature review, experimental feedback, debugging, reproduction checks, and preparation of figures, tables, and scripts.The authors state that AI tools did not operate experiments or implement the training method, and that they take responsibility for the final manuscript and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Sections 3 and 4, plus Appendices B.2 and B.3, describe the method and training and evaluation configurations.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 SENSITIVITY TO THE TOKEN-RESIDUAL SCALE

• 91.9% at α = 1, 89.1% at α = 2, and 79.8% at α = 3 are ALFWorld mean success rates; WebShop mean success rises from 59.7% to 65.7% and 73.8%.WebShop mean task score also rises from 72.5 to 80.1 and 83.9; these opposing trends suggest ALFWorld favors a smaller scale and WebShop a larger one among tested settings.Source: §A.1 · p. 12, Figure 5 · p. 12

A.2 TURN AND TOKEN CREDIT IN AN ALFWORLD TRAJECTORY

• 1.494, 2.348, and 1.875 are the positive turn advantages across the three turns; token residuals adjust credit within responses.For the pickup action, “take” receives +0.180 and “from” −0.695; in the final action, “desklamp” and “1” receive +0.421 and +0.441, yielding fused advantages of 2.296 and 2.316.Source: §A.2 · p. 13, Figure 6 · p. 13
• Displayed residuals need not average to zero because the figure omits thinking text; centering uses all valid generated tokens in the complete response.Source: §A.2 · p. 13, Figure 6 · p. 13

B.1 TASK INTERACTION AND REWARDS

• ALFWorld gives reward 10 for success and 0 otherwise, with an additional 0.1 penalty for invalid actions.Source: §B.1 · p. 13
• WebShop training rewards 10 only for a purchase ending with task score 1; other outcomes receive 0, and invalid actions incur an additional 0.1 penalty.Reported WebShop task scores are scaled to 0–100 for presentation; that scale does not change the environment score or training reward.Source: §B.1 · p. 13
• Each turn’s task reward and invalid-action penalty are placed at the final valid generated-token position, while KL regularization is applied separately to the actor loss.Source: §B.1 · p. 13

B.2 TRAINING SETTINGS

• Table 3 summarizes VETTA’s ALFWorld and WebShop training hyperparameters, including rollout budgets, PPO settings, learning rates, and update counts.Source: §B.2 · p. 14, Table 3 · p. 14

B.3 EVALUATION SETS AND AGGREGATION

• The fixed ALFWorld evaluation uses 140 seen and 134 unseen tasks, while the WebShop endpoint uses a fixed pool of 500 tasks.The locally evaluated results use three decoding seeds and report their arithmetic mean and sample standard deviation; some credit-composition and critic-structure results instead use one seed.Source: §B.3 · p. 14

B.4 CRITIC-DEPTH TIMING

• Critic-time measurements are within-run value-inference plus critic-update times, not end-to-end speedups; some depth pairs span different hosts.The table reports medians over 112 clean training steps.Source: §B.4 · p. 14, Table 4 · p. 15

C.1 VALUE-POSITION ALIGNMENT

• For response length L, aligned critic outputs precede each generated token; the turn target is supervised only at the first aligned position.Token targets use every valid response position, with causal attention preventing future tokens from changing earlier aligned values.Source: §C.1 · p. 15

C.2 VALID GENERATED-TOKEN POSITIONS

• The response mask includes generated positions through the first end-of-sequence token, excludes later padding, and selects token targets, centering positions, and actor-loss positions.Token-level GAE connects valid generated positions across responses of the same trajectory; each turn contributes one turn-value target at its first aligned position.Source: §C.2 · p. 15

C.3 CRITIC REGRESSION OBJECTIVE

• The two critic heads are reduced over their own valid positions before their losses are summed, without dividing the sparse turn loss by the token count.Source: §C.3 · p. 15

C.4 RESPONSE-WISE PROPERTIES OF ADVANTAGE FUSION

• The response-wise projection selects the token-credit vector closest to scaled token advantages while constraining its mean to the turn advantage.It characterizes scalar credit weights algebraically, not as an additional training optimization step.Source: §C.4 · pp. 15–16
• Under the stated special case, the common sampled return cancels from the within-action residual; the identity describes credit variation, not an individual token’s effect.Source: §C.4 · p. 16

C.5 ACTOR-LOSS IMPLEMENTATION

• The reported runs use dual clipping and KL regularization against a frozen reference policy, averaging both loss terms over valid generated-token positions.These settings are inherited from the token-wise PPO implementation shared with PPO and PPO*; they are not specific to VETTA’s value estimation or advantage fusion.Source: §C.5 · p. 16