Source-linked AI summary
EVOKE: Eliciting World Knowledge in Agents for Transferable Decision-Making
Yuhan Guo, Jinming Liu, Liang Xu, Ziqiang Li, Jianguo Huang, Zhicheng Wang, Hu Zhu, Qiuyu Chen, Yuntao Wei, Xin Jin, Wenjun Zeng
TL;DR
LLM agents often fail to transfer to unseen environments, while prediction-based world models add training cost and can compound planning errors. EVOKE elicits pretrained action knowledge by ranking actions under alternative goals at fixed states. Across several tasks and backbones, it improves task performance and unseen-environment generalization.
Problem
LLM agents often perform well in trained environments but transfer poorly to unseen ones, while learning action consequences through prediction adds training cost and can compound planning errors.
Source: §1 · p. 1Method
EVOKE holds state and history fixed while ranking the same candidate actions under alternative goals, eliciting pretrained knowledge of action consequences.
Source: §1 · p. 2Results
Takeaways & Limitations
For LLM agents, eliciting world knowledge through decision supervision offers a practical alternative to acquiring it through prediction.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly deployed as agents for multi-step decision-making, yet transfer poorly to unseen environments. World-model methods address this by training agents to predict future observations, at the cost of additional training and errors that compound when predictions are used for planning. However, for LLM agents operating in digital environments, much of this world knowledge is already internalized during pretraining, which shifts the problem from acquiring it to eliciting it. We argue that typical post-training provides little pressure for such elicitation, since supervision under a single goal at each visited state inadvertently drives policies to rely on superficial contextual habits. We introduce EVOKE, a post-training method that supplies this pressure through goal diversity at fixed states. Motivated by theory showing that an agent competent across diverse goals must encode a world model recoverable from its action preferences, EVOKE holds the environment state and interaction history fixed and ranks the same candidate actions under alternative goals, forcing action preferences to change, so that a policy relying on contextual habits or single-goal correlations cannot order them correctly. This implicitly elicits the policy's pretrained world knowledge to inform decisions. We evaluate EVOKE across diverse tasks in three backbones, demonstrating improved task performance, unseen environment generalization, and data efficiency. We further conduct controlled analyses to better understand what drives these gains. These findings offer a new perspective on eliciting internalized world knowledge for transferable action through direct decision supervision.
1 INTRODUCTION
LLM agents often fail to transfer to unseen environments, while prediction-based world models add training cost and can compound errors during planning. EVOKE instead uses goal diversity at fixed states to elicit pretrained action knowledge, and improves performance, generalization, and data efficiency across several task types.
Motivation
- LLM agents can perform well in training environments yet drop considerably on unseen ones, with learned behavior rarely transferring.Prediction-based world models add a training objective and can compound errors when predicted futures guide multi-step planning.Source: §1 · p. 1
- In digital environments, much action-effect knowledge may already be present from pretraining, shifting the challenge from acquiring it to eliciting it.Single-goal supervision can let policies memorize habitual actions for familiar contexts, providing little pressure to use that knowledge.Source: §1 · p. 2
EVOKE
- EVOKE ranks the same candidate actions under alternative goals at fixed states and histories, pressuring policies to use pretrained knowledge of action consequences.When action preferences differ across goals, contextual habits or single-goal correlations cannot rank the actions correctly.Source: §1 · p. 2
Evaluation
- Across household tasks, web navigation, and search-based QA, EVOKE improves task performance, unseen-environment generalization, and data efficiency.The authors also conduct controlled analyses to investigate what drives the gains.Source: §1 · p. 2
2 RELATED WORK
Related work studies world-model learning, policy optimization, and learning across goals. EVOKE differs by supervising preferences among actions at the same visited state under alternative goals.
World models for LLM agents
- World-model methods learn action consequences through explicit lookahead models or auxiliary prediction objectives, while WKM uses a separate knowledge model to guide planning.From Word to World reports that pretrained LLMs can predict next states in structured text environments from a few demonstrations; EVOKE elicits consequence knowledge without a prediction objective or separate model.Source: §2 · p. 3
Policy optimization for LLM agents
- Reinforcement learning, distillation, trajectory preferences, and DAgger vary how they compute training signals from agent interaction.EVOKE complements these methods by varying the goals under which each visited state is supervised.Source: §2 · p. 3
Learning across goals
- Goal-conditioned RL, successor features, hindsight relabeling, and language-agent relabeling share experience across goals in different ways.Richens et al. (2025) show that competence across a sufficiently rich set of goals entails a recoverable world model; EVOKE applies goal diversity to a single decision by ranking actions at fixed state and history.Source: §2 · p. 3
3 EVOKE
EVOKE trains a language-model policy using goal-conditioned action preferences at states collected from its own rollouts. It intervenes on goals while holding the state, history, and action set fixed, then assesses and ranks actions through an iterative training loop.
3 EVOKE
- EVOKE trains a policy from a goal, interaction history, and currently available actions, using state collection, goal intervention, action assessment, and contrastive ranking.At deployment it is a standard policy.Source: §3 · p. 3
3.1 GOAL INTERVENTIONS AT FIXED STATES
- Goal interventions keep the state, history, and candidate actions fixed while changing the goal, separating shared action consequences from goal-specific value.When preferred actions differ, a context-only mapping cannot rank them correctly under both goals.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
3.2 CONSTRUCTING GOAL-CONDITIONED ACTION PREFERENCES
- The method collects both progress states and policy detours or failures, then adds achievable alternative goals compatible with the unchanged scene.Actions are executed from the same state and labeled positive or competing based on observed outcomes; policy-favored mistakes are prioritized among competitors.Source: §3.2 · pp. 4–5
3.3 LEARNING THROUGH CONTRASTIVE RANKING
- Contrastive ranking raises probability mass on the positive set and separates each positive from competitors, including the policy’s tempting mistakes.Labels come from executed outcomes and annotator judgments, but those outcomes are not included in the policy input.Source: §3.3 · p. 5
3.4 ITERATIVE AGGREGATION
- Each training round repeats collection, goal intervention, action assessment, and hard-negative selection with the updated policy, aggregating data across rounds.At deployment, the policy uses only the goal, history, and available actions, with no world-model module, planning, or annotator.Source: §3.4 · p. 5
4 EXPERIMENTS
The experiments test whether goal-conditioned decision supervision improves complete task execution in unseen environments across three backbones, then analyze which training choices account for gains. EVOKE leads benchmark averages across all backbones.
4.1 SETUP AND EVALUATION PROTOCOL
- Evaluation covers ALFWorld success under a 50-action limit, seven-dataset search-QA accuracy, and WebShop task score and success rate.ALFWorld averages six task types; search-QA Avg equally weights seven datasets, while the main comparison covers three backbones.Source: §4.1 · p. 5, §4 · p. 5
- The comparisons span prompting, policy optimization and distillation, and world-model and dynamics methods.The three-backbone main comparison is followed by analyses of supervision design and training budget using a shared 3B initialization.Source: §4.1 · pp. 5–6, §4 · p. 5
4.2 MAIN RESULTS
- EVOKE achieves the best average on every benchmark and backbone, including world-model and dynamics methods on 7B, and remains best on unseen ALFWorld games.The margins are largest on multi-step ALFWorld and WebShop and on the smallest backbone, Qwen3-1.7B.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 6
5 ABLATIONS AND ANALYSIS
The analyses use a shared walkthrough-initialized policy and evaluate seen and unseen ALFWorld games to test which training choices account for EVOKE’s gains. They examine alternative-goal supervision, data efficiency, iteration, encoded action knowledge, goal-directed decisions, and execution efficiency.
5.1 DO THE GAINS COME FROM GOAL INTERVENTIONS?
- Unseen success falls from 91.8% with EVOKE to 82.8% with Source only, which uses the same ranking objective but removes alternative-goal contexts.Source replay reaches only 85.9% even when given the same number of updates as EVOKE, and Source only takes 40.7% more actions per game.Source: §5.1 · p. 7, Table 4 · p. 7
5.2 IS MORE GOAL DATA ENOUGH?
- EVOKE beat SFT in all 15 paired data-budget runs, and with 60% of labeled states already exceeded SFT trained on all states.Positive SFT receives the same goal contexts and positive actions but trails EVOKE by 7.5 points on unseen games; Figure 3 shows EVOKE ahead at every budget.Source: §5.2 · pp. 7–8, Figure 3 · p. 8
5.3 DOES ITERATION KEEP IMPROVING THE POLICY?
- Across three backbones, two aggregation rounds raise unseen success by 20.1–31.3 points, with gains in every round.On Qwen2.5-3B, the rounds supervise about 9.1% as many decisions as the demonstrations behind πboot.Source: §5.3 · p. 8, Figure 4 · p. 8
5.4 IS THE KNOWLEDGE ALREADY THERE?
- Action consequences exceed 97.0% probe accuracy on ambiguous transitions, yet unseen-game success is 60.4% for πboot and 91.8% after EVOKE.The representations are equally decodable in πboot and EVOKE, indicating that encoded consequences alone do not ensure the policy uses them to decide.Source: §5.4 · pp. 8–9, Figure 5 · p. 8
5.5 DOES EVOKE TURN THIS KNOWLEDGE INTO GOAL-DIRECTED DECISIONS?
- On 812 unseen-game goal pairs, πboot solves 19.5%, while EVOKE makes the fewest habitual errors at 6.4% of decisions.EVOKE makes 4.3 points fewer habitual errors than Positive SFT (95% CI [3.0, 5.6]); habitual errors choose the action correct for the other goal.Source: Figure 5 · p. 8, §5.5 · p. 9
5.6 BETTER DECISIONS OR MORE TRIAL AND ERROR?
- EVOKE solves 91.0% of unseen games within 20 actions, versus at most 80.6% for other trained variants, and needs 24.4–28.9% fewer actions per game.It also takes the fewest actions on the 70 games all five variants solve, makes less than half as many invalid actions as Positive SFT, and about a third as many revisits as other trained variants.Source: §5.6 · p. 9, Figure 6 · p. 9
6 CONCLUSION
EVOKE elicits pretrained world knowledge for transferable decisions by ranking actions under different goals at fixed states. Across three backbones and household, web-navigation, and search-based QA tasks, it improves task performance and unseen-environment generalization.
6 CONCLUSION
- EVOKE ranks actions under different goals at fixed states, eliciting pretrained world knowledge for transferable decisions.The method compares action preferences against the policy’s tempting mistakes.Source: §6 · p. 9
- Across household tasks, web navigation, and search-based QA with three backbones, EVOKE improves task performance and generalization to unseen environments.The analyses indicate that the pretrained backbone already represents action consequences and that EVOKE makes decisions depend on goals rather than habits.Source: §6 · p. 9
- An LLM assists with data construction by proposing alternative goals and assessing executed actions, but is not used at deployment.The authors state that they developed the research ideas, methods, and conclusions and take responsibility for the manuscript.Source: AI USE STATEMENT · p. 10
- The paper describes its methods, evaluation, and diagnostic analyses and says it will release code, checkpoints, and scripts for independent verification and extension.The benchmarks used in the experiments are publicly available.Source: REPRODUCIBILITY STATEMENT · p. 10
A IMPLEMENTATION DETAILS OF EVOKE
EVOKE initializes a policy from successful demonstrations, then gathers policy-visited states and supervises action rankings across achievable alternative goals. Its implementation uses executed outcomes to form preferences, applies iterative adapter training, and evaluates performance across environments under specified protocols.
A IMPLEMENTATION DETAILS OF EVOKE
- 1,482 decisions—about 9.1% of πboot’s 16,247 demonstration steps—supervise two rounds that raise unseen success from 60.4% to 91.8%.πboot is initialized by supervised fine-tuning on successful per-step demonstrations; states include progress and detours, repeated actions, or failures.Source: A IMPLEMENTATION DETAILS OF EVOKE · p. 15
- Alternative goals are retained when compatible with the scene, unsatisfied, and achievable, while state, history, and available actions remain fixed.The original task text is removed from the visible history so the policy sees only the current goal.Source: A IMPLEMENTATION DETAILS OF EVOKE · p. 15
- Candidates are executed from the same state and labeled by observed outcomes; preference groups pair up to four advancing actions with up to four competitors, prioritizing the policy’s greedy mistake.Uncertain candidates are discarded, and groups are weighted when a context has multiple groups.Source: A IMPLEMENTATION DETAILS OF EVOKE · p. 15
- Training averages action-completion token log-probabilities and applies a ranking loss while continuing LoRA adapters across rounds with the backbone frozen.The optimization uses τ = 1 and λ = 0.25; Table 5 lists the training schedule.Source: A IMPLEMENTATION DETAILS OF EVOKE · p. 15, Table 5 · p. 15
- The same pipeline applies to WebShop and search-based QA, with goals, states, and actions adapted to each environment.Labels use executed outcomes such as the next page or retrieved documents.Source: A IMPLEMENTATION DETAILS OF EVOKE · p. 16
B EVALUATION PROTOCOL
- Evaluation generates greedy free-text actions without legal-action masking or output repair, and reports micro and macro success using three-seed medians and game-resampled confidence intervals.An episode succeeds when task completion is reported within 50 actions; macro success is the unweighted mean over six task types.Source: B EVALUATION PROTOCOL · p. 16
B.1 DATA-EFFICIENCY PROTOCOL
The data-efficiency protocol compares nested labeled-state subsets under matched training conditions. It reports micro success across budgets and defines the unseen EVOKE-minus-SFT difference.
B.1 DATA-EFFICIENCY PROTOCOL
- Data-efficiency subsets use 20%, 40%, 60%, 80%, or 100% of labeled states and are nested, with all goal contexts and preference groups retained for each selected state.States are stratified by collection round and task type.Source: §B.1 · p. 16, Table 7 · p. 16
- Both objectives use the same subsets, two-round schedule, and three training orders, with update counts scaling to the amount of data.The schedule matches the full model, and the training orders are shared across objectives.Source: §B.1 · p. 16
- Table 8 reports micro success as mean ± s.d. over three training orders; its Δ is unseen EVOKE minus SFT.All budgets, including 100%, are retrained on nested subsets independently of Table 4.Source: Table 8 · p. 16
C ADDITIONAL RESULTS
The additional-results material presents iterative-training results and taskwise decision-supervision ablations. The supplied captions specify the reported metrics and evaluation splits, but not the result values.
C ADDITIONAL RESULTS
- Table 9 reports Seen and Unseen micro success after each iterative-training round.Source: Table 9 · p. 17
- Table 10 presents taskwise decision-supervision ablations as success percentages on Seen and Unseen games, averaged over training seeds.Source: Table 10 · p. 17
C.1 ADDITIONAL ABLATIONS
The additional ablations compare alternative-goal supervision and ranking competitors under controlled settings. Consequence probes test whether action outcomes are decodable from policy representations and compare against probe controls.
C.1 ADDITIONAL ABLATIONS
- EVOKE’s unseen-success difference over Source replay had a 95% confidence interval of [1.3, 9.5] under a fixed training budget.Both variants used the same 600 policy-visited states and 76 + 75 updates; half of EVOKE’s contexts replaced the original goal with an alternative while keeping state and history fixed.Source: §C.1 · p. 17, Table 11 · p. 17
- Competitor ablations kept EVOKE’s data, positives, and objective fixed while changing which actions were ranked below the positives.Alternatives included positives from other goals, random valid actions, and valid actions least likely under πboot.Source: §C.1 · p. 17, Table 12 · p. 17
D.1 CONSEQUENCE PROBES
- Consequence probes use held and contained transitions from unseen games to test whether policy hidden states predict post-action labels.Probe inputs include the goal, prior history, and candidate action; labels come from simulator outcomes, with action-text and unchanged-label baselines for comparison.Source: §D.1 · p. 18, Table 14 · p. 18
- Probe controls test whether outcome-label decodability depends on pretraining or exceeds memorization of action types and history text.Controls use random Qwen2.5-3B architecture weights, a random-label task, and a classifier trained on goal, history, and action text.Source: §D.1 · p. 19, Table 15 · p. 19
D.2 GOAL-CONDITIONED ACTION CHOICE
On unseen-game goal pairs, EVOKE improves goal-conditioned action choice over πboot, source replay, and positive SFT. Pair-level choice and whole-game success measure different abilities, with execution also affecting task success.
D.2 GOAL-CONDITIONED ACTION CHOICE
- EVOKE improves joint accuracy by 27.3 [23.6, 31.2], 22.4 [18.9, 26.0], and 3.2 [0.2, 6.2] points over πboot, source replay, and positive SFT, respectively.Intervals are 95% game-level bootstrap intervals.Source: §D.2 · p. 19
- EVOKE reduces habitual errors by 16.9 [15.0, 18.8], 6.9 [4.8, 9.0], and 4.3 [3.0, 5.6] points versus πboot, source replay, and positive SFT, respectively.A habitual error is choosing an action positive for the other goal in a pair.Source: §D.2 · p. 19
- Goal-pair accuracy isolates decisions, while unseen-game success also reflects execution: source replay has 4.2 vs. 11.9 invalid actions per game and 55.4% pair accuracy, unchanged from πboot.Positive SFT chooses by the goal more often but makes more invalid actions and revisits than source replay; EVOKE improves both.Source: §D.2 · p. 19