PaperlayerView exact revision on arXiv

Source-linked AI summary

World Potential Model: Pretrained World Knowledge as Progress Potentials

Jun Zhao, Jixin Tang, Yang Shu, Jinyang Wu, Yuyang Lu, Jingqi Tong, Hao Xu, Weifeng Ge, Qi Zhang

arXiv:2610.09560v1cs.AI

TL;DR

Long-horizon agents receive terminal outcomes that provide little information about intermediate progress, while task-specific evaluators can be costly to construct. This paper uses pretrained world knowledge to judge realized progress and anchors those judgments to task milestones; WPM-guided optimization improves success over outcome-only GRPO across evaluated configurations.

  • Problem

    Terminal outcomes provide little signal for distinguishing productive intermediate behavior from stagnation or regression, and constructing task-specific evaluators across heterogeneous tasks can be expensive and difficult to scale.

    Source: §1 · p. 2

  • Method

    A World Potential Model uses pretrained world knowledge to judge goal-conditioned realized progress, with task-specific milestone anchors converting judgments into scalar potentials for step-level signals.

    Source: §1 · p. 2

  • Results

    In ALFWorld and ScienceWorld, WPM-guided optimization improves success over outcome-only GRPO across evaluated settings, while pretrained models recover realized-progress structure above chance without evaluator fine-tuning.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The results provide initial evidence that realized-progress evaluation is reusable and can provide useful supervision for long-horizon agents.

    Source: §1 · p. 2

    Evaluation in broader tasks and agent settings remains an important direction.

    Source: §7 · p. 10

Abstract

from arXiv · show

Long-horizon language agents often receive supervision only from terminal task outcomes, leaving little signal for distinguishing productive intermediate behavior from stagnation or even regression. Rather than learning a separate value function or process reward model for every task, we ask whether pretrained models can recognize task progress from their existing world knowledge. We formalize this capability with a World Potential Model (WPM), a goal-conditioned evaluator of task-relative realized progress in agent contexts. In ALFWorld and ScienceWorld, off-the-shelf pretrained models substantially outperform chance at recovering realized-progress structure without task-specific evaluator fine-tuning. We further anchor these progress judgments to task-specific milestones to obtain scalar world potentials, whose temporal differences provide process-sensitive step-level credit for policy optimization. Under matched comparisons, WPM-guided optimization improves success over outcome-only GRPO across all evaluated configurations. Together, these results provide initial evidence that pretrained world knowledge can support reusable realized-progress evaluation and provide useful supervision for long-horizon agents.

1 Introduction

The paper asks whether pretrained world knowledge can identify realized progress in long-horizon tasks without task-specific evaluator training. It formalizes this as a World Potential Model and reports that its progress judgments support improved policy optimization.

1 Introduction

  • Terminal outcomes give little structure for distinguishing productive intermediate behavior from stagnation or regression, while task-specific evaluators can be costly to build across heterogeneous tasks.The paper asks whether pretrained knowledge can support progress evaluation without learning a separate evaluator for each task.Source: §1 · p. 2
  • The World Potential Model evaluates goal-relevant progress already realized in an agent context, using ordinal judgments and task-specific milestones to derive scalar potentials for step-level signals.Temporal differences between task-local potentials provide process-sensitive credit.Source: §1 · p. 2
  • In ALFWorld and ScienceWorld, pretrained models recover realized-progress structure above chance without evaluator fine-tuning, and WPM-guided optimization improves success over outcome-only GRPO across evaluated configurations.These findings provide initial evidence that realized-progress evaluation is reusable and useful for long-horizon agents.Source: §1 · p. 2

2 Problem Setup and World Potential Models

The paper models long-horizon tasks through evolving agent contexts with sparse terminal feedback, then defines retrospective, task-relative progress judgments and a milestone-based scalarization. It distinguishes this evaluator from predictors of future return, task-trained reward models, and environment-dynamics models.

2.1 Long-Horizon Sparse-Reward Agent Tasks

  • Terminal binary outcomes provide little direct information about goal-relevant progress at intermediate contexts, which may evolve through observations, actions, tool outputs, and other interaction history.The agent-visible context need not be Markov.Source: §2.1 · pp. 2–3

2.2 World Potential Models

  • A World Potential Model retrospectively judges task-relative realized progress, using ordinal comparisons between contexts rather than requiring globally calibrated numerical scores.The comparisons concern contexts from the same task instance; unrelated tasks need not share a numerical scale.Source: §2.2 · p. 3

2.3 Anchored Scalar World Potentials

  • For scalar supervision, ordered task-specific accomplishment anchors provide a local reference frame, and a WPM maps contexts onto that frame using pretrained knowledge.In the experiments, anchors are constructed once per task from its description and one successful expert trajectory, then fixed; that trajectory is withheld when new contexts are evaluated.Source: §2.3 · pp. 3–4

2.4 Relation to Existing Abstractions

  • Unlike a value function, a WPM evaluates realized progress rather than policy-dependent future return; unlike task-trained reward models, its evaluator receives no task-specific fine-tuning.Task-specific structure enters through the scalarization reference frame, while conventional world models predict dynamics rather than evaluate realized progress.Source: §2.4 · p. 4

3 Do Pretrained Models Expose World-Potential Structure?

The section tests whether pretrained models recover task-relative realized progress through pairwise comparisons and localization against task-specific milestones. Across ALFWorld and ScienceWorld, the models perform above chance on both evaluations without task-specific evaluator fine-tuning.

3 Do Pretrained Models Expose World-Potential Structure?

  • The study asks whether pretrained models can recover meaningful task-relative progress structure without task-specific evaluator fine-tuning, at both ordinal and anchored levels.Table 1 reports pairwise progress comparison and anchored progress localization.Source: §3 · p. 5, §3.3 · p. 6

3.1 Evaluation Tasks

  • Pairwise comparison ranks two contexts for the same goal without milestones or a shared trajectory prefix, while localization predicts a context’s index on a fixed task-specific milestone ladder.The ladder’s anchor coordinate supplies the corresponding scalar world potential.Source: §3.1 · p. 5

3.2 Experimental Setting

  • The evaluation uses seven language and vision-language models from three families, a shared prompting protocol, and environment-grounded labels that are withheld from the models.Labels use normalized remaining expert-plan length in ALFWorld and native partial-credit task scores in ScienceWorld; chance for localization varies with anchor count.Source: §3.2 · pp. 5–6

3.3 Results

  • Pairwise progress accuracy averages 92.52% across models and environments, with best results of 95.33% on ALFWorld and 98.00% on ScienceWorld, above the 50% random baseline.The result holds across three model families, language-only and vision-language models, and both environments.Source: §3.3 · p. 6
  • Localization accuracy averages 74.71%, with all models above the 20.00% ALFWorld and 15.27% ScienceWorld random baselines; best results are 88.00% and 85.33%, respectively.Localization is harder than pairwise comparison because it places contexts among multiple ordered, task-dependent anchors.Source: §3.3 · p. 6

4 Progress-Sensitive Policy Optimization with World Potentials

The section turns task-local world potentials into step-level credit for policy optimization. A process-sensitivity parameter preserves intermediate progress estimates in returns, which are then standardized across valid steps in a task’s rollout group.

4 Progress-Sensitive Policy Optimization with World Potentials

  • The method uses an anchored task-local scalar potential to convert changes in realized progress into process-sensitive step-level credit for critic-free policy optimization.The section tests this construction as supervision beyond sparse terminal outcomes.Source: §4 · p. 6

4.1 World-Potential Transition Signal

  • The temporal difference in world potential augments sparse environment reward, giving positive potential contribution for transitions to higher potential and negative contribution for transitions to lower potential.When potential remains unchanged and γ < 1, the potential contribution is a small negative signal.Source: §4.1 · pp. 6–7

4.2 Process-Sensitive Returns

  • The process-sensitivity parameter ρ controls how future intermediate contexts contribute to credit at step t; at ρ = 1, the intermediate terms cancel telescopically.At ρ = 0, the return reduces to the one-step transition signal.Source: §4.2 · p. 7
  • For 0 < ρ < 1, intermediate world potentials remain in the return, distinguishing trajectories with the same terminal outcome but different progress over time.Their contribution decays geometrically with temporal distance; smaller ρ emphasizes nearer contexts, while larger ρ extends credit farther into the trajectory.Source: §4.2 · p. 7

4.3 Step-Level Group-Relative Advantages

  • WPM-guided optimization assigns each valid environment step its own advantage from process-sensitive returns, unlike outcome-only GRPO’s single trajectory-level advantage shared across actions.Steps followed by progress, stagnation, or regression can therefore receive different optimization weights within one trajectory.Source: §4.3 · p. 8

5 A Utility Case Study in Sparse-Reward Policy Optimization

The study compares WPM-guided optimization with outcome-only GRPO under matched actor-training configurations in two sparse-reward environments. WPM-guided optimization improves success in all six actor–environment configurations.

5 A Utility Case Study in Sparse-Reward Policy Optimization

  • The controlled utility case study evaluates whether process-sensitive credit adds useful supervision beyond terminal outcomes, using task success as its criterion.The authors describe this as a controlled utility test, not a comprehensive comparison of reinforcement-learning algorithms.Source: §5 · p. 8

5.1 Experimental Setup

  • The comparison uses Qwen2.5-Instruct 3B and 7B and Qwen3 4B actors, with matched task sampling, rollouts, and actor updates across the two optimization methods.Both methods use the same three random seeds per actor–environment configuration; a frozen Qwen3-VL-32B evaluator supplies potentials for WPM-guided training.Source: §5.1 · p. 8

5.2 Results

  • WPM-guided optimization improves task success by 3.13 to 8.33 percentage points over matched outcome-only GRPO, averaging 5.08 points across all six actor–environment configurations.The largest gain is for Qwen2.5-7B on ScienceWorld, where success rises from 66.40% to 74.73%.Source: Table 2 · p. 9, §5.2 · p. 9

6 Related Work

The related work situates WPMs alongside intermediate supervision and pretrained-model reward or progress construction, while distinguishing their focus on retrospective ordinal judgments of realized progress.

Process supervision and progress-based credit

  • Prior interactive-agent process-supervision methods use rollout-derived credit, learned evaluators, intrinsic beliefs, or explicit subgoal structure; WPMs instead study eliciting realized-progress judgments from a frozen pretrained model without task-specific evaluator fine-tuning.The WPM’s primary interface is ordinal comparison rather than predicted future return.Source: §6 · p. 9

Foundation models for reward and progress construction

  • Other pretrained-model methods construct rewards or progress signals, while WPMs isolate ordinal judgments about progress already realized and use task-specific anchors only when scalar potentials are needed.For 0 < ρ < 1, intermediate potentials remain in the process-sensitive return, unlike classical policy-invariant potential-based shaping.Source: §6 · p. 9

7 Conclusion

WPM-guided optimization improves task success over outcome-only GRPO across evaluated settings, while pretrained models recover realized-progress structure and convert judgments into scalar potentials and step-level credit. The appendix describes the implementation interfaces and supporting evaluation details.

7 Conclusion

  • Under matched actor-training configurations, WPM-guided optimization improves task success over outcome-only GRPO across all evaluated settings.Source: §7 · p. 10
  • In ALFWorld and ScienceWorld, pretrained models recover substantial realized-progress structure without task-specific evaluator fine-tuning, then anchor judgments into scalar potentials and step-level credit.Source: §7 · p. 10
  • Evaluating world potentials across broader tasks and agent settings remains an important direction.Source: §7 · p. 10

A World Potential Model Instantiation and Capability Evaluation Details

A.1 WPM Progress-Judgment Interfaces

  • Pairwise comparison probes ordinal progress, whereas anchored localization maps a context to a task-local scalar potential using milestone anchors.Source: §A.1 · pp. 14–15

A.2 Task-Specific Anchors Construction

The experiments construct a fixed task-local milestone ladder from a successful expert trajectory, then use it as reference calibration for frozen WPM judgments on agent contexts.

A.2 Task-Specific Anchors Construction

  • A pretrained model receives the task description and one successful expert trajectory to construct reference milestones; the trajectory neither trains the evaluator nor accompanies later context evaluations.Source: §A.2 · p. 15
  • The fixed ladder supplies sparse reference points; the frozen WPM localizes previously unseen, potentially off-demonstration contexts against them.Source: §A.2 · p. 15
  • Milestone ladders contain 4–10 verifiable, state-based accomplishments with increasing coordinates reflecting relative progress, from 0 at initiation to 1 at completion.Source: §A.2 · pp. 15–16

A.3 Environment-Grounded Capability Evaluation Labels

The appendix constructs capability-evaluation labels from environment-grounded progress, then derives how temporal weighting of world potentials shapes step-level credit and trajectory preferences.

A.3 Environment-Grounded Capability Evaluation Labels

  • Privileged environment progress scores define evaluation targets, while ALFWorld uses a normalized planner-distance proxy and ScienceWorld uses its native task score.The ALFWorld proxy is not itself a definition of retrospective realized progress; the ScienceWorld score provides partial credit for satisfied goals and subgoals.Source: §A.3 · p. 17
  • Pairwise comparisons test the ordering between contexts, while anchored localization tests their positions relative to fixed task-specific progress anchors.Both use the same environment-grounded progress signal; equal-score pairs are discarded, and ALFWorld also excludes near-tied planner distances.Source: §A.3 · pp. 17–18

B.1 Return Decomposition

  • Equation (29) decomposes process-sensitive return into environment reward and current, intermediate, and terminal world-potential contributions.For 0 < 𝜌< 1, intermediate potentials contribute to step-level credit.Source: §B.1 · p. 19

B.2 Temporal Weighting Controlled by 𝜌

  • Smaller 𝜌 concentrates credit on near-term progress, while larger 𝜌 extends its temporal range and shifts relative emphasis toward environment rewards and terminal potential.At 𝜌=1, intermediate-potential contributions vanish; the parameter is not an optimal-setting formula because task horizon, rewards, and evaluator noise also matter.Source: §B.2 · pp. 19–20

B.3 Illustrative Trajectory Preferences

  • The trajectory examples hold environment return and selected endpoint potentials fixed to isolate how intermediate realized progress affects return.The comparisons illustrate earlier progress and preserved progress versus temporary regression followed by recovery.Source: §B.3 · pp. 20–21

B.3.1 Earlier Versus Later Realized Progress

  • For 0 < 𝜌< 1, the objective prefers realizing the same terminal progress earlier and preserving it; this preference weakens as 𝜌 increases and vanishes at 𝜌=1.Trajectory A realizes high potential earlier and preserves it, whereas Trajectory B reaches that potential only at termination.Source: §B.3.1 · p. 21

B.3.2 Preserved Progress Versus Intermediate Regression

  • For 0 < 𝜌< 1, preserving progress yields higher return than temporarily regressing and later recovering to the same terminal potential.The preference is zero at 𝜌=0 and 𝜌=1; sensitivity to progress or regression farther in the future requires a larger 𝜌.Source: §B.3.2 · pp. 21–22

B.4 Experimental Setting

The policy-optimization setting fixes 𝜌 across tasks and compares WPM-guided optimization with outcome-only GRPO under a shared actor-training pipeline.

B.4 Experimental Setting

  • This setting retains a nonzero intermediate-potential coefficient of 0.05 while allowing delayed progress and terminal outcomes to influence earlier actions.The appendix describes the setting as allowing substantially delayed signals to affect earlier actions.Source: §B.4 · p. 23
  • The experiments use 𝜌=0.95 uniformly across tasks, without per-task tuning or systematic sensitivity analysis.The authors treat it as a fixed algorithmic choice, not an empirically optimal or universally preferable setting.Source: §B.4 · p. 23

C Policy Optimization Implementation Details

  • WPM-guided optimization and outcome-only GRPO share task sampling, rollout construction, policy models, and actor-update configurations, differing in learning signal and credit construction.The implementation uses a shared actor-training pipeline.Source: C Policy Optimization Implementation… · p. 23

C.1 Environments and Task Sampling

The experiments use ALFWorld and ScienceWorld, sampling policy-training tasks and evaluating on the benchmarks’ official held-out partitions.

C.1 Environments and Task Sampling

  • ALFWorld covers compositional household navigation and manipulation, while ScienceWorld covers text-based scientific procedures and reasoning across 30 task types.Both require multi-step interaction with simulated environments.Source: §C.1 · p. 23
  • Policy training samples from each benchmark’s official training partition; evaluation uses ALFWorld’s 134 valid_unseen games and ScienceWorld’s official test variations.ScienceWorld development variations are not used in policy-optimization experiments.Source: §C.1 · p. 23
  • Trainer dataset records serve as rollout slots, not additional benchmark examples or a new split; task instances are sampled from the official partition at environment initialization or reset.Source: §C.1 · p. 23

C.2 Training Configuration and Software

The appendix specifies shared policy-training settings, how each method constructs its learning signal, the success-rate evaluation protocol, and artifact-use conditions.

C.2 Training Configuration and Software

  • Training uses three specified Qwen policy models, 16 task groups with 8 rollouts each per update, a 30-action rollout cap, and 100 epochs.Both methods use the same actor-training configuration, including a 1 × 10−6 learning rate and PPO-style clipped updates with KL regularization.Source: §C.2 · p. 23

C.3 Learning-Signal Construction

  • Outcome-only GRPO uses terminal binary outcomes for trajectory-level advantages, while WPM-guided optimization uses action-level potentials and process-sensitive returns for step-level advantages.Tokens from the same environment action share its step-level advantage; the remaining actor-update procedure is shared.Source: §C.3 · p. 24

C.4 Evaluation

  • Policies are evaluated by episode-level task success rate, counting an episode as successful when its final environment output reports info["won"] = True.The same evaluation protocol is used for vanilla, outcome-only GRPO, and WPM-guided policies.Source: §C.4 · p. 24

C.5 Artifact Licenses and Terms of Use

  • The experiments use released benchmarks and model checkpoints under their respective terms, limiting artifact use to research-oriented evaluation, inference, and policy optimization.The authors assume no broader licensing rights for benchmark assets distributed separately from their software repositories.Source: §C.5 · p. 24
Loading 2610.09560v1…