PaperlayerView exact revision on arXiv

Source-linked AI summary

Steering Black-Box Large Language Model Agents with Pitfall Anticipation World Model

Jiongxiao Wang, Han Ding, Youzhi Luo, Yawei Wang, Yun Zhou, Linbo Liu, Panpan Xu

arXiv:2610.29139v1cs.AI

TL;DR

Black-box LLM agents lack knowledge of how a specific environment responds, and prior world-model guidance leaves open what to predict and when to intervene. PAWM anticipates pitfalls and selectively provides bounded forward-looking guidance, with SFT followed by GRPO co-training. It outperforms baselines on both reported benchmarks, while evaluation remains scoped to two environments and primarily to Claude Sonnet 4.5 co-training.

  • Problem

    Black-box agents lack task-specific environment knowledge, and prior world-model work leaves unclear what guidance to provide and when to invoke it.

    Source: §1 · pp. 1–2

  • Method

    PAWM anticipates likely pitfalls and selectively gives multi-step guidance, trained with SFT on synthetic pitfall data and GRPO co-training with a frozen black-box agent.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    PAWM (+GRPO) reaches 78.9% Acc on ScienceWorld and 80.4% on WebArena-Lite, improving over the unassisted agent by 24.3% and 7.7%, respectively.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    Experiments show that anticipating pitfalls is an effective way for a world model to assist black-box LLM agents, with learned pitfall knowledge transferring across agent backbones.

    Source: §6 · p. 8

    Experiments are limited to ScienceWorld and WebArena-Lite and primarily use Claude Sonnet 4.5 for trajectory collection and GRPO co-training; agent-specific optimal anticipation remains underexplored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Frontier black-box Large Language Model (LLM) agents are proficient at long-horizon reasoning and precise action generation. However, when it comes to a specific agentic task, they have no internalized knowledge of how that particular environment unfolds. A world model offers a natural remedy, providing the agent with environment specific knowledge by foreseeing how the environment will respond to the action it is about to take. Prior world models for LLM agents mainly predict next observations at each step, which is costly and often uninformative because individual observations provide limited task-relevant guidance and not every step requires call for foresight. This raises a key question: what a world model should predict and when should it be invoked for agentic tasks? Our pilot study suggests that world model guidance is most useful when it anticipates pitfalls within a bounded multi-step future. Motivated by this, we introduce Pitfall Anticipation World Model (PAWM), which selectively intervenes when a potential pitfall arises and predicts the impending failure to steer the agent away from it. PAWM is trained in two stages: Supervised Fine-Tuning (SFT) on synthetic data labeled with agent pitfalls, followed by GRPO co-training with a black-box LLM to further optimize it in the target environment. PAWM improves Claude Sonnet 4.5 performance by 24.3% on ScienceWorld and 7.7% on WebArena-Lite, outperforming a range of baselines. These results demonstrate that PAWM provides an effective form of world model guidance for black-box LLM agents.

1 Introduction

Black-box LLM agents lack task-specific environment knowledge, while prior steering and world-model approaches leave open what guidance to provide and when. A pilot study motivates PAWM, which intervenes on anticipated pitfalls with bounded, forward-looking guidance.

1 Introduction

  • Black-box agents lack environment-specific knowledge, whereas prior steering often reinforces planning capabilities they already possess.Prior world models commonly simulate actions to next observations for ranking or search, leaving the usefulness of direct environment guidance unclear.Source: §1 · pp. 1–2
  • The paper frames guidance output and intervention timing as open design choices: forecasts can span different horizons, and guidance at every step may overwhelm or misdirect an agent.Intervening too rarely may also miss consequential steps.Source: §1 · p. 2
  • Pilot results indicate that guidance bounded at the next subgoal works best, while pitfall-related guidance dominates instances that change the agent’s action.Routing only pitfall-anticipating guidance nearly matches all-step guidance at far less cost.Source: §1 · p. 2
  • PAWM predicts impending failure and gives multi-step guidance only when a pitfall is likely, staying silent on promising trajectories.The intended intervention point is where the agent would otherwise fail.Source: §1 · p. 2
  • PAWM steers Claude Sonnet 4.5 to 78.9% on ScienceWorld and 80.4% on WebArena-Lite, improving over the unassisted agent by 24.3% and 7.7%, respectively.The two-stage approach uses SFT on synthetic pitfall data followed by GRPO co-training with the frozen black-box model.Source: §1 · p. 3

2 Related Work

Prior work steers black-box agents through external guidance and uses world models for several agent tasks. This paper studies continuous world-model assistance within the agent loop.

2 Related Work

  • Black-box steering methods use prompt optimization or auxiliary models that provide input-dependent guidance while keeping the underlying LLM fixed.Recent agentic-task advisors provide high-level guidance.Source: §2 · p. 3
  • World models for LLM agents have been used to generate synthetic experience, internalize dynamics, or support action selection through simulation and search.PAWM instead considers continuous assistance to a black-box LLM inside the agent loop.Source: §2 · p. 3

3 Pilot Study

The pilot study tests what guidance a world model should provide and when it should intervene for a fixed black-box agent. It identifies bounded, multi-step forecasts and potential pitfalls as useful design principles.

3 Pilot Study

  • The pilot study systematically varies forecast horizon and intervention timing for a fixed Claude Sonnet 4.5 agent on ScienceWorld and WebArena-Lite.The world model uses Qwen3.5-27B, so performance differences reflect its guidance rather than changes to the agent.Source: §3 · p. 3, servable Markov Decision Process (POMDP) · p. 4

3.1 Problem Definition and Experiment Setup

  • The task formalization keeps the black-box policy frozen and aims to improve its performance by learning a world model.The agent proposes an action, receives predicted-consequence guidance, then reproposes an action; only that revised action is executed.Source: servable Markov Decision Process (POMDP) · p. 3

3.2 What Should the World Model Output?

  • SubGoal-Traj performs best on both benchmarks, while next-observation guidance yields marginal gains and full-trajectory guidance falls below baseline on WebArena-Lite.The study also finds that the highest action-change rates do not ensure the highest task accuracy.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 When Should the World Model Intervene?

  • Across both benchmarks, guidance that changes the tentative action is associated with anticipated failure or stagnation, and pitfall anticipation provides a potential intervention criterion.The pilot's resulting design principles are to forecast multiple steps to a meaningful milestone and intervene around potential pitfalls.Source: §3.3 · p. 5
  • Pitfall-only gating nearly matches passing all guidance while using substantially fewer black-box API calls; non-pitfall-only guidance performs worse despite intervening more often.The study identifies pitfall anticipation as a promising intervention criterion that retains most guidance benefit at lower cost.Source: §3.3 · p. 5, Table 3 · p. 5

4 Pitfall Anticipation World Models

PAWM monitors a frozen black-box agent and intervenes when it anticipates an impending failure, offering forward-looking guidance. It is defined and trained through SFT on synthetic pitfall data followed by GRPO co-training with the agent.

4.1 PAWM Definition

  • PAWM gates intervention: it emits <pass> when no pitfall is anticipated, otherwise describes the impending failure and guides the agent to repropose its action.The gate controls when guidance is injected; its content provides multi-step, forward-looking guidance concentrated at pitfalls.Source: §4.1 · pp. 5–6

4.2 Stage 1: SFT on Synthetic Pitfall Dataset

  • SFT labels sampled successful and failed trajectories with pitfall steps and corrective guidance, using contrasts where available and alternate supervision when runs lack mixed outcomes.For all-success tasks, the annotator identifies residual pitfalls; for all-failure tasks, ground-truth guidance is injected to obtain successful trajectories for comparison. Pitfalls are described using only observations available at that step, and <pass> steps are down-sampled to an empirically selected ratio.Source: §4.2 · p. 6

4.3 Stage 2: GRPO Co-Training with the Black-box LLM Agent

  • GRPO co-trains PAWM in the black-box agent loop, refining its anticipations using outcome-based rewards rather than only agreement with annotator labels.For each task, the method samples a group of trajectories containing generated pitfall guidance and optimizes PAWM.Source: §4.3 · p. 6

5 Experiment

Experiments evaluate PAWM against steering baselines on WebArena-Lite and ScienceWorld, finding the strongest results for PAWM with GRPO and improved accuracy across tested agent backbones. A pass-to-anticipation ratio study also shows that training-label balance affects intervention rate and accuracy.

5.1 Experimental Setup

  • Evaluations use 165 verified WebArena-Lite tasks and 271 ScienceWorld test tasks, with results averaged over three runs.Claude Sonnet 4.5 is the agent by default, and world models use a Qwen3.5-27B backbone.Source: §5.1 · p. 7
  • Baselines include an unassisted agent, retrieval and pre-task guidance, and world-model guidance with different output contents.GRPO is also applied to baseline steering methods on ScienceWorld; on WebArena-Lite it is run only on PAWM because of higher cost and slower online interaction.Source: §5.1 · p. 7, §5.2 · p. 7

5.2 Results

  • 78.9% Acc on ScienceWorld and 80.4% Acc on WebArena-Lite; PAWM (+GRPO) improves over N/A by 24.3% and 7.7%, respectively.PAWM (+GRPO) performs best across both benchmarks and metrics; GRPO improves Acc more than Pass@3.Source: Table 4 · p. 7, §5.2 · p. 7
  • On ScienceWorld, GRPO improves over SFT across the various steering methods.Source: Figure 2 · p. 8
  • PAWM consistently improves accuracy across tested Claude, GLM 5, and Kimi K2.5 agent backbones.The authors report that learned pitfall knowledge is not tied to the agent used for co-training.Source: Figure 3 · p. 8, §5.2 · p. 8
  • A larger <pass>-to-anticipation training ratio consistently lowers evaluation Intervention Rate; best accuracy occurs at 1.0 on ScienceWorld and 0.75 on WebArena-Lite.These SFT checkpoints are used as the final SFT models and GRPO initialization.Source: §5.2 · p. 8, Table 5 · p. 8

6 Conclusion

The paper presents PAWM, which anticipates likely pitfalls to steer black-box LLM agents, and trains it with SFT followed by GRPO co-training. Experiments report that it outperforms baselines and transfers learned pitfall knowledge across agent backbones.

  • PAWM anticipates likely pitfalls to steer black-box LLM agents, with SFT on synthetic pitfall data followed by GRPO co-training with the frozen agent.Source: §6 · p. 8
  • PAWM-steered Claude Sonnet 4.5 outperforms various baselines, and its learned pitfall knowledge transfers across agent backbones.Source: §6 · p. 8
  • The paper concludes that anticipating pitfalls is an effective form of world-model assistance for black-box LLM agents.Source: §6 · p. 8

Limitations

The study is constrained by its agent, benchmark, and task-specific training settings, as well as reliance on synthetic pitfall annotations.

Limitations

  • Experiments primarily use Claude Sonnet 4.5 for trajectory collection and GRPO co-training, leaving agent-specific differences in optimal anticipation underexplored.PAWM transfers to other agent backbones during evaluation, but they are not independently co-trained.Source: Limitations · p. 9
  • The evaluation covers only ScienceWorld and WebArena-Lite, while costly online interaction limits reinforcement-learning scale, particularly on WebArena-Lite.Source: Limitations · p. 9
  • The world models are trained on trajectories from a particular agentic environment, so conclusions may not extend to general-purpose models trained across domains.Source: Limitations · p. 9
  • PAWM uses hindsight-generated synthetic pitfall supervision that may contain noise or teacher-model bias and may miss some suboptimal behaviors.Source: Limitations · p. 9

Ethical Considerations

The authors note that inaccurate guidance could lead to undesirable behavior and that capability enhancement could be misused; evaluation is limited to benchmarks.

Ethical Considerations

  • Because evaluation is benchmark-limited, the authors urge caution in real-world deployment, where inaccurate guidance or misuse could have undesirable consequences.Source: Ethical Considerations · p. 9

A Related Works

The related-work discussion positions world models as environment-specific guidance for fixed black-box agents. The appendices describe turn-wise prompts for ScienceWorld and WebArena, including agent context and guidance for reconsidering proposed actions.

A Related Works

  • Unlike prompt optimization or high-level advisors, the paper frames its advisor as a world model supplying environment-specific knowledge to a fixed black-box agent.Prior world-model work also uses synthetic experience, internalizes dynamics, or jointly evolves models; other black-box settings use candidate simulation or search.Source: A Related Works · p. 12

B Prompts for Turn-wise Agent

B.1 ScienceWorld

  • ScienceWorld prompts provide the agent with location, room objects and states, inventory, the latest observation, and currently available actions.Source: §B.1 · pp. 12–13

System Prompt

  • The ScienceWorld system prompt requires one valid action copied verbatim, with grouped action lists expanded into concrete choices.Source: System Prompt · p. 13

User Prompt

  • The ScienceWorld user prompt supplies the task history, current location, inventory, observation, and grouped valid actions.Source: User Prompt · p. 13

Action Reproposal Prompt after User Prompt

  • Action-reproposal prompts distinguish a one-step observation forecast from multi-step trajectory forecasts, asking the agent to keep or revise its tentative action based on the prediction.The multi-step variants forecast either to the next milestone or to task completion.Source: Action Reproposal Prompt after User… · pp. 13–14

B.2 WebArena

  • WebArena prompts use a simplified HTML accessibility-tree-like page representation with semantic element IDs; the agent summarizes the page and plan in its history before choosing an action.Source: §B.2 · p. 14, User Prompt · p. 15

C Experiment Details of Pilot Study

This section provides additional experimental details for the pilot study.

C Experiment Details of Pilot Study

C.1 SFT Details for World Models

This section describes the SFT materials for world models in ScienceWorld and WebArena, including sampled trajectories, prompts, and targets for predicting immediate outcomes or bounded future paths. It also covers full-trajectory and milestone-bounded variants and reports the SFT model and optimization settings.

C.1 SFT Details for World Models

  • 2,124 ScienceWorld and 1,941 WebArena trajectories were collected by running Claude Sonnet 4.5 three times per task, and each trajectory was split at every step for target generation.Failed runs were retained because they show how poor actions produce unfavorable observations; for WebArena-Lite Next-Obv, the next-step summary already produced by the agent serves as the target.Source: §C.1 · p. 16

System Prompt for Next-Obv WM Targets Generation on ScienceWorld

System Prompt for Full-Traj & SubGoal-Traj WM Targets Generation on ScienceWorld

  • ScienceWorld trajectory targets describe the future as a coherent path, faithfully reflecting success or failure rather than transcribing each step or inventing events.Full-Traj targets continue through termination, while SubGoal-Traj targets stop at the next progress milestone or a clear failure; later predictions should be appropriately tentative.Source: System Prompt for Full-Traj &… · pp. 17–18, User Prompt for Full-Traj &… · p. 18

System Prompt for Full-Traj & SubGoal-Traj WM Targets Generation on WebArena

  • WebArena trajectory targets chain the proposed action’s resulting page state with every subsequent action and observation in actual order, without skipping or inventing steps.They must reflect the real outcome, avoid revealing future-specific values not already known from the task or history, and stop at the specified termination, horizon, or milestone.Source: System Prompt for Full-Traj &… · pp. 18–19, User Prompt for Full-Traj &… · p. 19

User Prompt for Full-Traj & SubGoal-Traj Targets Generation on WebArena

  • The WebArena target-generation input includes the task objective, agent history, actual future, run outcome, and an instruction defining where the summary should stop.The listed world-model input prompts are also used at inference time.Source: User Prompt for Full-Traj &… · p. 19

System Prompt for Next-Obv WM on ScienceWorld

User Prompt for Next-Obv WM on ScienceWorld

  • The ScienceWorld next-observation prompt supplies the current state and proposed action, then requests the actual outcome exactly one step later inside anticipation tags.The forecast describes the resulting game response and state change, whether the action advances the task or fails.Source: User Prompt for Next-Obv WM on… · p. 20

System Prompt for Full-Traj WM on ScienceWorld

  • The ScienceWorld full-trajectory prompt forecasts from the proposed action through task termination, predicting success or failure rather than assuming success.It asks for a compact, increasingly tentative path grounded in the current state, history, and environment mechanics.Source: System Prompt for Full-Traj WM on… · pp. 20–21

System Prompt for SubGoal-Traj WM on ScienceWorld

  • The ScienceWorld subgoal prompt defines a milestone as the next meaningful progress step, then forecasts only until it is reached or the attempt clearly fails.Milestones may take one or several steps; if the milestone is task completion, the forecast continues through termination.Source: System Prompt for SubGoal-Traj WM on… · p. 21

User Prompt for Full-Traj & SubGoal-Traj WM on ScienceWorld

  • The ScienceWorld user prompt provides agent history, current state, and proposed action, then instantiates either a full-trajectory or next-milestone forecasting instruction.Both variants request the forecast in anticipation tags.Source: User Prompt for Full-Traj &… · pp. 21–22

System Prompt for Next-Obv WM on WebArena

  • The WebArena next-observation prompt predicts the actual browser page immediately after each proposed action, including productive, failed, or unchanged outcomes.It conditions the prediction on the browser state, site mechanics, and agent history, and requests a brief neutral description.Source: System Prompt for Next-Obv WM on… · p. 22

User Prompt for Next-Obv WM on WebArena

  • The WebArena user prompt supplies the task objective, agent history, current browser state, and proposed action, then requests the actual next page observation in anticipation tags.The page forecast includes its URL or page and key visible information.Source: User Prompt for Next-Obv WM on WebArena · p. 22

System Prompt for Full-Traj WM on WebArena

  • The WebArena full-trajectory prompt forecasts the path from the proposed action through task termination, including likely success or failure and uncertainty farther into the future.It requests a compact, state-grounded forecast rather than a per-click script.Source: System Prompt for Full-Traj WM on… · p. 23

User Prompt for Full-Traj & SubGoal-Traj WM on WebArena

  • Qwen3.5-27B was fine-tuned for 5 epochs with a global batch size of 128, learning rate 2e-6, and a 20,480-token cap on 16 NVIDIA B200 GPUs.Training used LlamaFactory, cosine learning-rate decay, and a 0.1 warmup ratio.Source: User Prompt for Full-Traj &… · p. 24

C.2 Lexical Analysis Details

The lexical analysis compares guidance that changed tentative actions with guidance that did not, using weighted log-odds z-scores to identify distinguishing words. The top-ranked words in the CHANGED group describe anticipated failure or stagnation.

C.2 Lexical Analysis Details

  • Counts are smoothed with α, with α0 = α |V|, before comparing word frequencies between CHANGED and UNCHANGED groups.The log-odds ratio measures how much more frequent a word is in CHANGED than in UNCHANGED.Source: §C.2 · p. 24
  • The analysis ranks words by weighted log-odds z-scores, computed separately for each benchmark.The z-score down-weights rare terms whose estimates carry high variance.Source: §C.2 · p. 24
  • The most distinctive CHANGED-group words describe anticipated failure or stagnation.The analysis presents the top-20 words in Table 6 and visualizes the result in Figure 4.Source: §C.2 · p. 24

C.3 Semantic Analysis Details

The semantic analysis uses a prompt to classify whether world-model guidance anticipates a pitfall. The system prompt defines the failure-anticipation criterion and the user prompt supplies the guidance for a YES-or-NO judgment.

C.3 Semantic Analysis Details

  • The appendix presents the prompt used to classify whether world-model generations anticipate a pitfall.It is used in the semantic analysis described in Section 3.3.Source: §C.3 · p. 24

System Prompt for Pitfall Anticipation Classification

  • The system prompt counts warnings that an action or current course will fail, stall, loop, be ineffective, or miss the goal as failure anticipation.Descriptions, confirmations, and next-step suggestions without an implication of failure are not classified as anticipation.Source: System Prompt for Pitfall Anticipation… · p. 24

User Prompt for Pitfall Anticipation Classification

C.4 Gating Ablation Details

This appendix details the gating classifier and synthetic-label prompts used to train PAWM to forecast action-caused failures or avoidable detours, with forecasts tailored to each environment and grounded in information available at the proposed action. It also documents the WebArena RL setup, including measures addressing shared-state contamination and infrastructure and evaluation changes.

C.4 Gating Ablation Details

  • The gating ablation uses an LLM classifier to decide whether SubGoal-Traj guidance is pitfall-related, and reproposes the action only when it is.Claude Sonnet 4.5 serves as the classifier, with domain-specific failure examples added to its system prompt for each benchmark while keeping its decision criterion unchanged.Source: §C.4 · pp. 24–25

System Prompt for Pitfall Anticipation Classification on ScienceWorld

  • The classification prompts label guidance failure-anticipatory when it warns of failure, wasted or roundabout steps, wrong targets, or unverified completion, but not when it merely describes a state or sound action.The ScienceWorld prompt classifies a forecast, while the WebArena prompt classifies advisor guidance; both require exactly one yes/no tag.Source: System Prompt for Pitfall Anticipation… · p. 25

D Prompts for Pitfall Labeling and PAWM Inputs

System Prompt for Contrastive Labeling on ScienceWorld

  • For ScienceWorld contrastive labeling, the model locates the earliest fixable action that starts an unrecoverable failure and forecasts its consequences from the state then visible.It first checks for environmental noise, which yields no pitfall; the forecast must predict rather than command and avoid hindsight-specific measurements or answers.Source: System Prompt for Contrastive Labeling… · p. 26

User Prompt for Contrastive Labeling on ScienceWorld

  • The ScienceWorld contrastive user prompt supplies the task and successful and failed trajectories, then requests the noise check and forecast for the failure’s start step.The output is a single pitfall object, or none if the noise check applies.Source: User Prompt for Contrastive Labeling… · pp. 26–27

System Prompt for Success-only Labeling on ScienceWorld

  • For ScienceWorld success-only labeling, the model forecasts avoidable detours in a successful run, flagging only waste that was avoidable from the actor’s knowledge at that step.Necessary exploration is not a detour, and clean runs yield an empty array; each forecast identifies the detour’s start and end steps and predicts wasted work without hindsight.Source: System Prompt for Success-only… · p. 27, User Prompt for Success-only Labeling… · p. 27

System Prompt for Contrastive Labeling on WebArena

  • For WebArena contrastive labeling, the model finds the earliest action causing an unrecoverable failure and forecasts its consequences using only information visible at that step.A noise check excludes failures where the runs reach essentially the same state by the same productive path and differ only in the final reported answer.Source: System Prompt for Contrastive Labeling… · p. 28

System Prompt for Success-only Labeling on WebArena

  • For WebArena success-only labeling, the model forecasts genuine avoidable detours in successful runs and leaves clean runs unflagged.It identifies the start and end of wasted work, requires the forecast to be anticipatable from the actor’s state, and excludes correct termination; the user prompt provides the task and trajectory and requests an empty array when no detour exists.Source: System Prompt for Success-only… · pp. 28–29, User Prompt for Success-only Labeling… · p. 29

System Prompt for PAWM on ScienceWorld

  • PAWM’s ScienceWorld system prompt calls for a multi-step rollout and a brief forecast only when it reveals a materially useful outcome; routine actions pass silently.Examples include actions that fail to advance the task, violate preconditions, repeat an ineffective move, or irreversibly commit to the wrong target.Source: System Prompt for PAWM on ScienceWorld · pp. 29–30

User Prompt for PAWM on ScienceWorld

System Prompt for PAWM on WebArena

  • The WebArena system prompt flags likely immediate or downstream pitfalls, previews their consequences neutrally, and passes sound actions without commentary.It treats rejected actions, wrong targets or formats, lost progress, repeated failures, and unverified termination as potential pitfalls.Source: System Prompt for PAWM on WebArena · pp. 30–31

User Prompt for PAWM on WebArena

E RL Environment for WebArena

  • WebArena’s shared write-state risks cross-task contamination, so the setup uses isolated Incus containers for write-action sites, but remains fragile and costly.The authors report frequent timeouts and retries, with RL training costing roughly twice as much as ScienceWorld; they also add an HTML/accessibility-tree parser that roughly halves observation tokens, an LLM reward judge, and a 20k-character step-observation limit.Source: E RL Environment for WebArena · pp. 31–32

F Training Details for PAWM

PAWM is trained with SFT on synthetic pitfall data, then GRPO co-training in the target environments. The GRPO setup addresses sparse learning signals from tasks already solved by the initial agent.

F Training Details for PAWM

  • SFT fine-tunes Qwen3.5-27B on the synthetic pitfall dataset for five epochs, using batch size 128 and learning rate 2e-6.Training uses a cosine schedule, a 0.1 warmup ratio, and a 20480-token sequence cap.Source: F Training Details for PAWM · p. 32
  • GRPO uses dynamic sampling to discard rollout groups with identical rewards, which otherwise provide zero advantage and no learning signal.For WebArena-Lite, tasks solved in all three SFT-PAWM attempts are also filtered out, leaving 375 training tasks.Source: F Training Details for PAWM · p. 32
  • Both benchmark runs start from their corresponding SFT PAWM, use shared GRPO hyperparameters, and train on 16 NVIDIA B200 GPUs.Table 7 lists the GRPO co-training hyperparameters.Source: F Training Details for PAWM · p. 32, Table 7 · p. 32

G Details of Baseline Methods

The baseline set includes an unsteered agent, retrieval- and pre-task-guidance methods, and world-model variants that intervene at every step. The pre-task prompts ask for reusable high-level plans formed before environment exploration.

G Details of Baseline Methods

  • Baselines include vanilla execution, retrieval of a successful training trajectory, pre-task guidance distilled from successful trajectories, and three world models that intervene at each step.RAG retrieves the most similar successful training instruction by embedding similarity; the world-model baselines vary the guidance content.Source: G Details of Baseline Methods · p. 32, User Prompt for Pre-Task Guidance · p. 34

System Prompt for Pre-Task Guidance on ScienceWorld

  • ScienceWorld pre-task guidance is a concise, self-contained plan formed from the task before room exploration, leaving the actor to adapt it to discovered objects and rooms.The prompt requests the shape of the solution rather than turn-by-turn actions or specific rooms.Source: System Prompt for Pre-Task Guidance on… · p. 33

System Prompt for Pre-Task Guidance on WebArena

  • WebArena pre-task guidance is a reusable, high-level plan formed from the objective before browsing, with the actor expected to explore pages and elements itself.The prompt asks for a promising direction rather than a literal click path or specific URLs.Source: System Prompt for Pre-Task Guidance on… · p. 33

User Prompt for Pre-Task Guidance

H Qualitative Examples for PAWM

Figures 5 and 6 show ScienceWorld cases where PAWM anticipates pitfalls that trap the baseline agent and steers it toward success; Figure 7 adds a WebArena-Lite example.

H Qualitative Examples for PAWM

Related papers

Loading 2610.29139v1…