PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Value Functions for Credit Assignment in Agentic RL

Kaijie Zhu, Weixiang Yan, Omar Attia, Abbas Kazerouni, Alexander Toshev, Andrew Szot

arXiv:2610.15840v1cs.LGcs.AI

TL;DR

Long-horizon LLM agents receive sparse outcome rewards, leaving action-level credit difficult to assign. The paper uses a frozen LLM to estimate policy-aware values for step-level RL credit and gated teacher supervision. Across agentic benchmarks, LVRL improves over GRPO and PPO.

  • Problem

    Long-horizon LLM agents receive sparse, often binary end-of-episode rewards, making it difficult to distribute feedback across individual actions; existing approaches trade coarse trajectory credit against the cost of learning a value function.

    Source: §1 · p. 1

  • Method

    LVRL uses a frozen LLM conditioned on current-policy rollout context and outcome-anchored endpoint values to derive step advantages, while LVOPD gates teacher supervision to actions with negative estimated value changes.

    Source: §1 · p. 2

  • Results

    LVRL consistently improves over GRPO and PPO across the evaluated benchmarks; on Qwen3-8B, AppWorld success is 26.3% versus 19.8% for GRPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The frozen critic redistributes sparse outcome rewards across steps to distinguish progress, mistakes, and recoveries without additional environment interactions.

    Source: §5 · p. 9

    Trajectory evaluation with Claude Opus 4.6 costs approximately 9k–20k input tokens and 0.6k–1.9k output tokens per target trajectory, depending on environment context length.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult. Methods such as GRPO assign the same trajectory-level advantage to all actions, while critic-based methods require learning an additional value function that can be costly and unstable. We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic. LVRL conditions the value model on group context and calibrates its predictions using known values at the initial and terminal states, enabling step-level credit assignment from estimated value differences. We further extend this idea to learning from an expert policy through on-policy distillation (OPD), introducing LLM Value Function OPD (LVOPD), which focuses teacher supervision on actions associated with decreases in estimated value. We evaluate our approach across diverse agentic benchmarks: AppWorld, BabyAI, and SWE-bench Verified. Across policy scales (4B to 32B), LVRL consistently and substantially outperforms both GRPO and PPO. On AppWorld and SWE-bench Verified, LVRL yields absolute improvements of up to 13.6 and 8.6 percentage points over GRPO. Notably, on tasks requiring extended exploration like BabyAI, LVRL bridges the gap from total failure to 75.0% success. Furthermore, LVOPD yields consistent improvements over standard distillation baselines on AppWorld. Finally, Monte-Carlo analysis confirms that our frozen critic accurately identifies semantic pitfalls and recoveries, proving that LVRL extracts robust, policy-aware step-level credit from sparse terminal outcomes.

1 Introduction

Long-horizon LLM agents receive sparse outcome rewards, making it difficult to assign credit to individual actions. The paper introduces frozen-LLM value estimation for step-level credit in RL and selective teacher supervision in distillation, and reports improvements across agentic benchmarks.

1 Introduction

  • The paper frames temporal credit assignment as distributing a single end-of-episode success-or-failure signal across individual actions in long-horizon agent tasks.Source: §1 · p. 1
  • LVRL uses a frozen, policy-aware LLM critic to turn sparse outcomes into dense per-step credit without updating a separate critic.It conditions value estimates on other rollouts and their outcomes, then anchors initial and terminal values to observed data so step advantages sum to the trajectory’s group-relative return.Source: §1 · p. 2
  • LVOPD gates teacher supervision to actions associated with decreases in estimated value, aiming to correct execution errors while preserving the student’s independent progress.Uniform distillation can penalize valid alternative strategies even when they differ from the teacher’s behavior.Source: §1 · p. 2
  • LVRL improves over GRPO and PPO across AppWorld, BabyAI, and SWE-bench Verified; on Qwen3-8B, AppWorld success is 26.3% versus 19.8% for GRPO.On BabyAI out-of-distribution scenarios, LVRL reaches 65.5% success while baseline methods fail entirely.Source: §1 · p. 2
  • Monte-Carlo analysis finds that predicted value changes identify 72% of critical pitfalls and 66% of recoveries; ablations highlight the importance of in-context policy calibration and critic capability.Source: §1 · p. 2

2 Related Work

Prior approaches provide trajectory-level credit, learn or estimate step-level values, or use auxiliary models and proxy signals. LVRL instead uses current-policy rollout context and outcome anchoring, while LVOPD uses estimated value decreases to gate teacher supervision.

2 Related Work

  • GRPO assigns a uniform advantage across a trajectory, while actor–critic methods learn a value function and Monte-Carlo methods estimate values through sampled continuations or shared-state returns.Other approaches use process reward models or implicit signals such as hindsight judgments, target-answer likelihoods, policy-reference log-ratios, or attention patterns.Source: §2 · p. 2
  • Unlike policy-agnostic progress estimation, LVRL conditions a frozen LLM on current-policy rollouts and anchors predictions to observed outcomes.The related work passage identifies GVL as a conceptually related method that estimates progress from robot videos using a frozen vision-language model.Source: §2 · p. 3
  • LVOPD gates reverse-KL distillation using negative consecutive value differences, so actions with non-negative predicted changes receive no distillation loss.This contrasts with standard token-level distillation, which can penalize a student for valid alternative strategies; the framework overview shows the gate applied after step advantages are computed.Source: §2 · p. 3, Figure 1 · p. 3

3 Method

The method addresses binary terminal rewards in finite-horizon agent tasks by replacing uniform trajectory credit with value-difference advantages. It also uses those advantages to gate on-policy teacher supervision.

3.1 Preliminaries

  • Agent tasks are modeled as finite-horizon MDPs with binary terminal success rewards; PPO learns a value estimator, whereas GRPO assigns each trajectory’s uniform group-relative advantage to every action.The uniform signal can reinforce mistakes in successful trajectories and penalize useful actions in failed trajectories.Source: §3.1 · p. 4

3.2 LLM Value Function RL

  • LVRL estimates intermediate values with a frozen LLM conditioned on other current-policy rollouts, anchoring initial values to group success rate and terminal values to observed rewards.Consecutive value differences form step advantages; endpoint anchoring makes their unnormalized sum equal the trajectory’s outcome advantage, so critic error redistributes credit within a trajectory without changing its overall advantage sign.Source: §3.2 · pp. 4–5

3.3 LLM Value Function OPD

  • LVOPD combines the clipped policy surrogate with reverse-KL teacher distillation weighted by the negative part of each step advantage.Actions with non-negative estimated value changes receive no distillation loss, while the gate scales supervision by the magnitude of the predicted value drop.Source: §3.3 · p. 5

4 Experiments

Across three agent benchmarks, LVRL improves long-horizon learning while LVOPD improves AppWorld distillation; validation and ablations examine value-signal accuracy and critic dependence.

4.1 Setup

  • Training spans AppWorld, BabyAI, and R2E-Gym with sparse terminal rewards; evaluation uses AppWorld Test Challenge, BabyAI layouts, and SWE-bench Verified.Policies include Qwen3-4B, Qwen3-8B, and Qwen3-32B; R2E-Gym training uses one seed because of higher computational cost.Source: §4.1 · pp. 5–6

4.2 Does LVRL Improve Long-Horizon Agent Learning?

  • LVRL consistently outperforms GRPO and PPO across domains and scales; Qwen3-8B reaches 26.3% success on AppWorld and 20.2% on SWE-bench Verified.Trajectory evaluation costs approximately 9k–20k input tokens and 0.6k–1.9k output tokens per target trajectory, depending on environment context length.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • On BabyAI, where GRPO and PPO score near 0%, LVRL reaches 75.0% ID and 65.5% OOD success by producing step advantages even when all group trajectories fail.Interior value estimates can vary despite zero endpoint anchors, assigning positive or negative advantages within failed trajectories.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Does LVOPD Improve On-Policy Distillation?

  • On AppWorld, LVOPD reaches 20.2% success with Qwen3-4B versus 17.0% for Pure OPD and 19.2% for Trajectory-gated OPD.For Qwen3-8B, LVOPD reaches 23.3% versus 22.2% for Pure OPD and 21.2% for Trajectory-gated OPD; LVOPD surpasses Pure OPD after about 100 training steps.Source: §4.3 · p. 7, Table 2 · p. 8
  • LVOPD’s efficacy is limited by critic capability; all evaluated OPD variants, including LVOPD, yield near-0% success on BabyAI.The Qwen3-235B critic used for LVOPD is weaker than Claude Opus 4.6 in the reported ablations.Source: §4.3 · pp. 7–8

4.4 Validating Step-Level Value Signals

  • Predicted value changes match Monte-Carlo change signs for 72% of pitfalls and 66% of recoveries.Agreement rises from 44% in the first trajectory third to 88% in the final third; in 12 of 19 early disagreements, the critic’s value estimate does not decrease while empirical Monte-Carlo value does.Source: §4.4 · pp. 8–9

4.5 Ablation Studies

  • Removing rollout-group calibration lowers AppWorld success from 26.3% to 22.1%, though the ablation remains above GRPO’s 19.8%.The full method achieves the highest mean success rate; endpoint anchors and same-task rollout summaries are removed together in this ablation.Source: Table 3 · p. 8, §4.5 · p. 9
  • On AppWorld, Qwen3-4B reaches 9.8% success with Qwen3-235B as critic, 22.3% with GLM-5.2, and 24.3% with Claude Opus 4.6.GLM-5.2 and Claude Opus 4.6 provide comparably high performance, suggesting LVRL generalizes across these frontier model families.Source: Table 4 · p. 8, §4.5 · p. 9

5 Conclusion and Discussion

LVRL redistributes terminal rewards into step-level credit using a frozen, policy-aware LLM, while LVOPD gates teacher supervision toward actions associated with decreasing estimated value. The conclusion reports that LVRL outperforms GRPO and PPO across agent environments, while noting critic capability and inference cost as constraints.

5 Conclusion and Discussion

  • LVRL anchors frozen-LLM state estimates to group success rates and terminal outcomes, then uses consecutive value differences for step-level credit; LVOPD gates supervision toward value-decreasing actions.Across tool use, software engineering, and spatial navigation, LVRL outperforms GRPO and PPO, while LVOPD improves on uniform and trajectory-gated distillation.Source: §5 · p. 9
  • LVRL’s gains depend on the frozen critic’s capability, and querying a large model for every trajectory can impose prohibitive inference overhead under constrained compute or throughput.The method is designed for multi-turn agent environments; extending it to non-agentic domains remains open.Source: §5 · pp. 9–10

A.1 AppWorld

  • AppWorld agents use a persistent Python REPL to discover APIs, retrieve credentials, and execute multi-turn tasks across simulated everyday applications.The environment awards binary reward only when its validation suite passes after completion or turn-budget exhaustion; declaring completion alone is insufficient.Source: §A.1 · p. 13
  • AppWorld optimization uses 90 training tasks, with final evaluation on 417 test_challenge tasks whose scenarios are disjoint from training.Evaluation uses three rollouts per task and reports pass@1.Source: §A.1 · p. 13

A.2 BabyAI Task Construction and Evaluation

The BabyAI benchmark varies navigation horizon while holding its task interface and object inventory fixed, and evaluates generalization to longer unseen routes. The accompanying R2E-Gym setup tests repository-level issue resolution on codebases disjoint from training.

A.2 BabyAI Task Construction and Evaluation

  • BabyAI uses a six-room, 2 × 3 grid with closed doors and 18 objects; instructions identify one target, while door connectivity and object placement vary across layouts.The agent receives textual egocentric observations and chooses one primitive action per turn.Source: §A.2 · pp. 14–15
  • Door distance is the number of doors on the shortest valid route to a cell adjacent to the target; generated distances of four and five define the out-of-distribution split.The training and in-distribution tasks cover distances one through three, and all splits use disjoint layouts.Source: §A.2 · pp. 14–15
  • BabyAI evaluation gives each policy 128 steps, with success measured by picking up the target within that budget; the OOD pool tests longer horizons under the same task definition and interaction rules.Reward is sparse and binary, granted when the target is picked up.Source: §A.2 · pp. 15–16

A.3 R2E-Gym and SWE-bench Verified

  • R2E-Gym presents a natural-language issue and containerized Python repository; agents edit the code and receive reward only if the held-out test suite passes.Training uses 3,200 instances from repositories disjoint from the 500-task SWE-bench Verified evaluation set, which measures transfer to unseen codebases.Source: §A.3 · pp. 16–17
  • SWE-bench Verified agents interact through tool calls in a fresh container, with a 50-turn episode limit and three evaluation rollouts over all 500 tasks.The evaluation uses the same harness and budgets across compared methods.Source: §A.3 · p. 17

B Training Details

The training details describe the estimated cost of value queries, the shared setup for LVRL baselines, and the frozen models and gated objective used for LVOPD. The value-query cost estimates exclude rollout-summarization calls.

B.1 Cost of Value Estimation

  • Value-estimation queries are estimated to cost $0.061–$0.062 per AppWorld trajectory, $0.107 for BabyAI, and $0.140 for R2E-Gym.Token counts use cl100k_base as an uncalibrated proxy, and these estimates exclude separate trajectory-summarization calls and caching or batch discounts.Source: §B.1 · p. 19

B.2 LVRL

  • LVRL, GRPO, and PPO share initial checkpoints, task sampling, rollout budgets, optimization updates, and generation settings for each environment–model pair.Policies are fine-tuned from Qwen3 checkpoints; training uses eight rollout groups of size eight for AppWorld and BabyAI.Source: §B.2 · p. 19

B.3 LVOPD

  • LVOPD uses Qwen3-235B as both frozen value estimator and teacher; normalized consecutive value differences set a nonnegative gate that weights teacher supervision on environment actions.The implementation approximates reverse KL on the student’s top-16-token support using teacher probabilities returned for the top 32 tokens.Source: §B.3 · pp. 19–20

C Prompts for Rollout Summarization and Value Estimation

  • The per-step rollout-summarization prompt converts a completed rollout into a concise action–result description for each step.The prompt is used for compression in the per_step mode.Source: Figure 6 · p. 21

D Training Curves

The reported training curves show faster LVRL learning than GRPO and PPO across all three environments, while LVOPD overtakes pure OPD after roughly 100 training steps.

D Training Curves

  • LVRL learns faster than GRPO and PPO in all three environments, while LVOPD overtakes pure OPD after roughly 100 training steps.The training-curve figures provide the visual comparison for these learning trajectories.Source: D Training Curves · p. 21

E LVOPD BabyAI Results

On BabyAI, all three OPD methods remain near 0% success on ID and OOD splits, while gated LVOPD variants outperform the OPD baseline on AppWorld after roughly 100 training steps. The Monte-Carlo protocol uses independent continuations and a conservative event threshold.

E LVOPD BabyAI Results

  • All three OPD methods remain near 0% success on both BabyAI ID and OOD splits.AppWorld reward curves show both gated variants outperforming the OPD baseline after roughly 100 training steps.Source: E LVOPD BabyAI Results · p. 22, Figure 10 · p. 23

F.1 Monte-Carlo Labeling Protocol

  • For AppWorld Monte-Carlo labels, 32 independent continuations from each recorded state estimate value as the success fraction.States are reconstructed by replaying the recorded prefix, and transitions with divergent replay are dropped before labeling.Source: §F.1 · p. 23

F.2 Choice of the Event Threshold

  • Table 9 reports the worst-case null tail probability and estimated spurious-event count for differences between estimates based on 32 samples.The final column estimates spurious events over the evaluation set under the null.Source: Table 9 · p. 24
  • The difference between two independent Monte-Carlo value estimates has bounded standard deviation; events require a change of at least 3σ.At 3σ, expected noise-induced events are small relative to the 114 observed, so the threshold yields a conservative event set.Source: §F.2 · pp. 23–24

F.3 Breakdown by Trajectory Position

Predicted-value agreement with Monte-Carlo events rises later in trajectories, while task-difficulty strata do not explain the position-dependent pattern. Qualitative cases show both a semantic pitfall despite successful execution and a missed downstream consequence.

F.3 Breakdown by Trajectory Position

  • Predicted-value changes agree with 70.2% of 114 selected Monte-Carlo events overall and 88.5% of 26 events in the final third.None of the selected events is a final transition, so terminal-value anchoring does not affect the reported agreement.Source: §F.3 · p. 24
  • Zero predictions fall from 29% in the first trajectory third to 4% in the final third, while median predicted magnitude rises from 0.025 to 0.100.Agreement increases across segments; the median Monte-Carlo magnitude increases more moderately.Source: §F.3 · p. 25
  • Opposite-sign rates are 12% on Easy, 16% on Medium, and 11% on Hard tasks, with no evidence that standard difficulty measures explain the position pattern.Rates are also similar across reference-solution lengths: 13% for at most 40 lines and 16% for longer solutions.Source: §F.3 · p. 25
  • Trajectory position is only a coarse proxy for distance from observable consequences, so the pattern does not establish long-range attribution as its cause.The examples illustrate local evidence the model uses successfully and downstream effects it can miss.Source: §F.3 · p. 25

F.4 Qualitative Examples

  • A payment request missing a required final period is a pitfall despite successful execution, and both value estimates identify the transition as severe.The requests cannot be un-created, and the mismatch likely causes failure; Table 11 summarizes three transitions, with the first two tracking Monte-Carlo values and the third moving in the wrong direction.Source: §F.4.1 · p. 25, Table 11 · p. 25

F.4.2 Recovery despite an execution error

A failed API call can still mark recovery when it fixes a key classification rule, while successful local data retrieval can conceal a downstream schema mismatch. The training examples are illustrative rather than part of the Monte-Carlo event counts.

F.4.2 Recovery despite an execution error

  • The agent correctly classifies files by date but uses the wrong API parameter, and the value model treats this execution-error step as a recovery.The call fails, but the parameter is described as trivial to repair; an execution-error rule would mislabel the transition because the state is not less likely to succeed.Source: §F.4.2 · p. 26

F.4.3 Missed pitfall with a downstream consequence

  • The agent verifies retrieval calls but not whether their outputs share a schema, mistaking local data collection for progress.This failure mode is consistent with lower agreement earlier in trajectories, where consequences are less explicit when actions are taken.Source: §F.4.3 · p. 26

G Examples of Training Trajectories

  • The three training cases are illustrative and have no Monte-Carlo labels, so they are excluded from the event counts.Their value differences are shown as interior, unnormalized differences.Source: G Examples of Training Trajectories · p. 27

G.1 AppWorld: a semantic error despite successful execution

The training examples show value changes tracking task semantics rather than execution success alone: deleting alarms instead of disabling them causes a sharp predicted drop, while freeing BabyAI inventory enables a later pickup.

G.1 AppWorld: a semantic error despite successful execution

  • Deleting six alarms instead of disabling them changes predicted value from 0.90 to 0.00, a −0.90 step, despite successful calls.Updating the remaining sleep alarm does not restore the deleted records, and the predictor places the drop at the semantic error.Source: §G.1 · p. 27

G.2 BabyAI: resolving an inventory bottleneck

  • In BabyAI, dropping the key into a free cell raises predicted value from 0.12 to 0.90, a +0.78 change, and frees inventory for the target pickup.Earlier ineffective attempts receive negative changes despite the eventual successful outcome.Source: §G.2 · p. 28

TASK AND OUTCOME

The agent fixes a paste-size issue by replacing an incorrect zero-area paste region with source-image dimensions. The reproduction then exits normally, and the value predictor assigns opposite signs to the two edits.

TASK AND OUTCOME

  • The task is to fix Image.paste so a 50 × 50 source can be pasted into a 100 × 100 destination without an explicit box; the 15-action trajectory ends with reward 1.The issue description is abridged.Source: TASK AND OUTCOME · p. 28
  • At step 13, the source-sized edit raises predicted value from 0.45 to 0.80 (+0.35); rerunning the reproduction at step 14 raises it to 0.95 (+0.15).The same reproduction exits normally at step 14.Source: TASK AND OUTCOME · p. 29
  • The first edit sets the paste box to zero area, and the reproduction still fails with an image-size error.The table records a predicted-value decrease from 0.70 to 0.35 at step 10, followed by a small increase after the failed rerun.Source: TASK AND OUTCOME · p. 29
  • Although both edits are accepted by the tool, the predictor gives them opposite signs, distinguishing patch semantics from tool-call validity without establishing correctness on untested inputs.The second edit uses source-image dimensions and the same reproduction exits normally.Source: TASK AND OUTCOME · p. 29

Related papers

Loading 2610.15840v1…