PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Agent-World Gap: Text World Models for LLM-based Agents

Yixia Li, Hongru Wang, Peng Lai, Zhiwen Ruan, He Zhu, Youxin Zhu, Ganlong Zhao, Minda Hu, Yun Chen, Sibei Yang, Peng Li, Jeff Z. Pan, Jia Pan, Guanhua Chen, Yang Liu, Guanbin Li

arXiv:2606.09032v1cs.CL

TL;DR

LLM-based agents often act reactively without an explicit model of how textual environments evolve. This survey formalizes and organizes text world models across their construction, applications, and evaluation, showing that simulator fidelity is central to their reliability and use.

  • Problem

    Many LLM-based agents lack an explicit, inspectable model of textual environments as structured, evolving systems.

  • Method

    The survey unifies text world models through a formal framework and agent-lifecycle review spanning foundations, construction, applications, and evaluation.

  • Results

    The review distinguishes LLM-as-WM and Code-as-WM approaches and shows that world-model uses and evaluations ultimately depend on prediction quality and simulator fidelity.

  • Takeaways & Limitations

    Simulator faithfulness is a first-class concern because world models support agent training, inference, and evaluation across dynamic interactive settings.

  • Takeaways & Limitations

    World-model applications are bounded by how accurately the simulator approximates the real environment, and retrieval-based approaches also depend on useful prior experience.

Abstract

from arXiv · show

Large language model (LLM)-based agents are increasingly used in interactive textual environments, from web navigation and code editing to tool use and long-horizon dialogue. Yet many remain largely reactive, mapping observations to actions without an explicit model of how these environments are structured and evolve. This motivates text world models (TWMs): transition models over textual states that, given a state and a candidate action, predict the resulting webpage, terminal output, API response, or user reply, thereby supporting planning, efficient learning, and principled evaluation. We systematically review text world models for LLM-based agents, organized around a formal framework and the agent lifecycle: (1) Foundations, defining text world models and characterizing them by state representation and grounding domain; (2) Construction, taxonomizing LLM-as-WM and code-as-WM paradigms and reviewing methods for building them; (3) Application, examining how world models support agents at training time through experience synthesis and at inference time through planning, verification, and adaptation; and (4) Evaluation, covering both evaluation of the world model itself and its use as an evaluation environment for agents. We aim to consolidate this rapidly developing area, clarify its design space, and highlight open challenges for future research.

1 Introduction

Text world models address the limitations of reactive LLM agents by modeling transitions over textual environments. This survey organizes the field around their foundations, construction, usage, and evaluation across the agent lifecycle.

  • Motivation: Many LLM agents act reactively from observations to actions without an explicit, inspectable model of structured and evolving environments.This limitation affects agents that browse the web, edit code, orchestrate tools, or conduct long-horizon dialogue.
  • Foundations: Text world models predict resulting textual states from a current textual state and candidate action across webpages, repositories, APIs, terminals, and dialogue.Textual state includes structured task states, observable traces, and natural-language environment descriptions.
  • Challenges: Text environments create open-vocabulary, knowledge-laden, and semantically subjective dynamics, complicating construction, use, and evaluation.Correctness often lacks a single right successor because plausible transitions depend on background facts.
  • Survey Scope: The survey addresses fragmented literature across web, code, tool-use, and dialogue by providing a systematic review of text world models as a distinct object.Existing world-model surveys primarily target visual and control regimes rather than textual environments.
  • Survey Organization: Its lifecycle framework covers formal foundations, learning-, prompt-, and programmatic construction, training- and inference-time usage, and evaluation from accuracy to agent utility.The framework compares methods by how they are built, deployed, and evaluated.

2 Foundations and Formalism

This section defines text world models as textual transition models and establishes their scope. It then organizes them by state/transition representation and grounding domain.

  • 2.1 Definition and Scope: Text world models are defined at the transition-function level, mapping an agent-visible state and candidate action to a textual rendering of the successor state.The rendering may be a natural-language description, structured record, code fragment, or other artifact specifying what holds after the action.
  • 2.1 Definition and Scope: “Text” describes how simulated transitions are exposed to the agent, while inputs may be textual or multimodal, including dialogue histories, terminal traces, and webpage screenshots.The supplied definition distinguishes the output form from the modality of the input state.
  • 2.1 Definition and Scope: Unlike latent-space world models and ordinary language models, text world models expose counterfactual state–action dynamics as text that can be read, edited, verified, and fed back into an LLM agent.Their role is explicitly counterfactual: they predict how an interaction state would change if an action were taken.
  • 2.1 Definition and Scope: The survey covers environment modeling of web interfaces, software repositories, and tool APIs, alongside user modeling of preferences, intentions, and task progress during dialogue.Environment dynamics appear as page updates, file changes, or deployed-runtime state transitions, whereas user modeling tracks human-side interaction variables.
  • 2.2 A Two-Axis Taxonomy: The first taxonomy axis spans natural language, structured representations, and executable code, ranging from flexible descriptions to formal operational encodings.Structured formats include JSON, key–value stores, knowledge graphs, accessibility trees, and PDDL-style predicates; executable formats include Python, TypeScript, HTML, simulators, and planning operators.
  • 2.2 A Two-Axis Taxonomy: Natural-language representations are easy for LLMs to produce and consume but weakly constrain consistency, completeness, and executability.This limitation contrasts with structured representations, which make entities, relations, preconditions, and effects easier to track and verify.
  • 2.2 A Two-Axis Taxonomy: Executable encodings can replace free-form prediction with execution when domains admit precise operational semantics, yielding stronger reproducibility and constraint enforcement.The encoding may represent part or all of a transition.
  • 2.2 A Two-Axis Taxonomy: The second taxonomy axis distinguishes physical, digital, social, and abstract worlds according to the regularities governing their dynamics.These include embodied or simulated physics; deployed computational systems; human or human-like behavior; and formal or symbolic rules in planning, mathematics, game theory, and logic.

3 Building Text World Models

Text world models are built through two paradigms distinguished by where state transitions are carried out: within an LLM’s forward pass or in code. LLM-based approaches obtain the transition function by updating model parameters on trajectory data or shaping the input context of a frozen model.

  • 3 Building Text World Models: Construction methods divide into LLM-as-world-model and code-as-world-model paradigms according to where the state transition is carried out.In the LLM-as-world-model paradigm, the transition function is the LLM’s own forward pass.
  • 3 Building Text World Models: LLM-based approaches obtain the forward pass by updating parameters on trajectory data or shaping the input context of a frozen model.These approaches are organized as §3.1 and §3.2, respectively.

3.1 Learning-Based Construction

Learning-based text world models are constructed by choosing prediction targets, sourcing trajectory data, and scaling collection to the environment’s openness, with these axes tightly coupled. Training objectives range from full-state or delta prediction under SFT to increasingly downstream-oriented RL rewards, although world-model RL remains less mature.

  • Prediction targets: World models predict either complete next states or action-induced deltas, with the appropriate target depending on observation structure and output scale.Full-state prediction suits compact, structured environments, whereas delta prediction reduces output space for large, redundant web or GUI observations; hybrid designs use both.
  • Trajectory data collection: Trajectory sources trade off realism and cost: real-environment rollouts provide in-distribution evidence, while synthetic or repurposed data is cheaper but potentially less faithful.Self-play and large-scale harvesting amortize collection costs through automation, whereas fully synthetic data inherits the LLM’s existing coverage of target dynamics.
  • Data scale: Data requirements increase with environment openness: ALFWorld and SciWorld saturate at ∼20k trajectories, WebShop improves at 70k, and StableToolBench remains unsaturated at 160k.These scaling patterns show that closed, structured environments need far less data than open-ended or tool-use environments.
  • Design trade-offs: The three construction axes are coupled: delta prediction is most useful for large, redundant observations, where real-environment data collection is also most expensive.Fully synthetic or repurposed data is cheapest but is most defensible when the prediction target is already well covered by the LLM.
  • RL-based training: RL-based construction defines rewards by prediction quality, progressing from surface fidelity to semantic, behavioral, and latent consistency, but remains less mature than SFT.Behavioral consistency addresses cases where textual or semantic similarity misses decision-critical information, while latent consistency removes dependence on recorded states, pretrained judges, or reference agents.

3.2 Prompt-Based Construction

Prompt-based construction builds text world models without parameter updates by prompting frozen LLMs, grounding predictions in external knowledge, or accumulating experience through interaction. These approaches trade prior-only simplicity against domain coverage, retrieval dependence, and the costs of exploration and distillation.

  • 3.2.1 In-Context World Modeling: Prior-only prompting is limited by the LLM’s domain knowledge, with errors compounding as planning horizons grow.LLMs exceed 75% on next-state identification but rarely exceed 65% on full-procedure planning alignment.
  • 3.2.2 Retrieval-Augmented World Knowledge: Retrieval augmentation conditions predictions on relevant external evidence, including procedural traces, tutorials, expert sequences, or step-level abstractions.R-WoM retrieves and reranks tutorials for single-pass long-CoT trajectory imagination, whereas TRAD retrieves annotated expert-step thoughts and addresses temporal mismatch.
  • 3.2.2 Retrieval-Augmented World Knowledge: Structured retrieval compresses past experience into reusable knowledge, such as causal feasibility rules, procedural heuristics, tuple-indexed experience, or external affordances.WorldMind builds its knowledge base through Predict-Act-Verify, storing failure-induced constraints and distilling successful trajectories.
  • 3.2.2 Retrieval-Augmented World Knowledge: Retrieving full traces favors concrete in-distribution fidelity, whereas distilled rules or prototypes generalize more broadly but depend on faithful distillation.Both retrieval directions assume useful prior experience, creating a cold-start limitation.
  • 3.2.3 Self-Evolving Prompt World Models: Self-evolving prompt world models learn from exploration or execution by recording outcomes, distilling reusable knowledge, and feeding it into later prompts.This approach replaces reliance on a fixed corpus or frozen prior with interaction-derived knowledge.
  • 3.2.3 Self-Evolving Prompt World Models: A short pre-deployment exploration episode can distill state-transition triples into causal rules that remain fixed across subsequent tasks without parameter updates.The exploration cost is paid once per environment, recovering much of finetuning’s benefit according to the supplied passage.

3.3 Programmatic Construction: Code as World Model

Code-as-world-model methods use LLMs to generate executable artifacts whose transitions can be run, inspected, replayed, and verified independently. This enables scalable environment synthesis, but usefulness depends on functional correctness, diversity, and domain-specific verification.

  • Construction paradigm: Code-as-world-model methods execute transitions deterministically, separating code generation by the LLM from independently running, reproducible, and verifiable world-model artifacts.This contrasts with predicting transitions through an LLM forward pass.
  • Artifact forms: Generated artifacts range from renderers such as HTML and screenshots to symbolic state machines such as FSMs and PDDL operators, and full simulator code.TheoryCoder combines PDDL operators for high-level structure with LLM-synthesized Python functions for low-level transitions.
  • Scaling environments: Environment count becomes a scaling dimension, but each synthesized environment must be functionally correct and the collection diverse enough to prevent policy collapse.The construction challenge therefore shifts from producing more environments to keeping the environment set useful.
  • Adaptive difficulty and diversity: RLVE establishes environment-count scaling across 400 environments by increasing difficulty when accuracy exceeds 90%, while AutoEnv generates rule-heterogeneous environments through three abstraction layers.These methods show that diversity complements raw environment count in making scaling effective.
  • Limitations: Count and diversity are complementary: raw count yields log-linear gains that saturate without diversity, while verification remains siloed because correctness criteria are domain-specific.Examples include SQL pre/post-conditions, Docker exit codes, and procedural unit tests, with no transferable standard for synthesized-environment correctness.

3.4 Cross-Paradigm Comparison

The choice between code-as-WM and LLM-as-WM depends on whether target dynamics are closed-form and codifiable or open-ended and knowledge-intensive. Across paradigms, methods trade fidelity, adaptability, and scalability against verifiability, calibration, and long-horizon consistency, while supervision and scaling increasingly emphasize rewards, evolving knowledge, and environment diversity.

  • When to use which paradigm: Code-as-WM fits fixed, codifiable dynamics and provides deterministic, replayable, and verifiable transitions, whereas open-ended, knowledge-intensive dynamics require LLM-as-WM.Examples of codifiable dynamics include game rules, tool APIs, and GUI state machines; open-ended cases include commonsense and longtail web behavior.
  • Strengths and limitations: Learning-based methods offer high fidelity and compression but sacrifice verifiability and long-horizon consistency; prompt-based methods adapt quickly but suffer hallucination and poor calibration.Code-as-WM provides deterministic, verifiable, and reusable transitions, but the passage truncates before stating its full cost.
  • Current trends: Supervision is shifting from token-level fidelity toward reward-based training and evolving knowledge stores, while scaling increasingly prioritizes synthesizing diverse environments.The passage identifies environment count and diversity as first-class scaling axes.

4 Training-Time World Models … 4.3 User Simulation for Agent Training

Training-time world models support agents by internalizing anticipation, synthesizing or simulating environments with increasing coupling, and modeling users for interactive learning. Across these settings, tighter integration improves alignment or personalization but exposes persistent fidelity and hallucination-drift challenges.

  • 4 Training-Time World Models: Training-time world models serve three roles: internalizing anticipation in agent parameters, fabricating training environments, and simulating users for interactive learning.The section shifts focus from constructing world models to their roles inside the agent training loop.
  • 4.1 Internalizing World Models into Agent Parameters: Internalizing world models avoids extra inference calls, supports co-adaptation with decision-making, and transfers anticipation to settings without test-time simulators.Methods differ in whether the internalized model remains implicit as a warm-start or is surfaced in reasoning traces.
  • 4.1 Internalizing World Models into Agent Parameters: Warm-start internalization improves training optimization and inference efficiency, whereas reasoning-trace simulation couples prediction with action selection but requires additional optimization machinery and inference-time tokens.The two approaches localize world-model benefits to training or to action-time decision-making.
  • 4.2 World Models as Training Environments: World models become increasingly coupled to training through offline trajectory synthesis, online rollout interaction, and co-evolution with the policy.Offline methods generate and filter fixed trajectories, online methods respond at every rollout step, and co-evolution updates simulator and policy together.
  • 4.2 World Models as Training Environments: Tighter coupling improves distribution alignment but raises training cost, while free-form simulators across regimes remain vulnerable to hallucination drift on out-of-distribution states.Mitigations re-ground simulators in structured databases or code and constrain their output spaces.
  • 4.3 User Simulation for Agent Training: User simulation is a distinct training paradigm because users are stochastic, preference-driven, and underspecified, requiring specialized modeling and evaluation choices.Simulated-user RL ranges from cooperative task-oriented users to vague, emotional, and persona-driven users, with increasingly specialized reward design.
  • 4.3.1 RL with Simulated User Environments: UserRL combines turn-level and trajectory-level reward shaping with an SFT cold-start, enabling an open-source mid-sized model to surpass proprietary baselines in human-confirmed tests.Its standardized environments cover intent clarification, persuasion, travel planning, and tool-calling.
  • 4.3.2 User-Model Fidelity and Personalization: User-model fidelity is a central bottleneck: dedicated user models expose weaknesses masked by cooperative simulators, while adaptation methods use clarification and post-action memory to track preferences.The trajectory progresses from simulation-only training through fidelity-aware modeling toward direct learning from live users, but simulation-based fidelity benchmarks often rely on other simulators.

4.4 Summary and Comparative Analysis

The three paradigms address distinct training-loop needs: retaining environment knowledge, substituting costly real-environment access, and simulating users. Across them, world models are shifting toward dynamic partners while sim-to-real and coverage drift remain recurring failures.

  • Three roles in the training loop: Internalization retains environment knowledge after training, whereas world-model-as-environment substitutes for limited or costly real-environment access during training.The latter spans one-shot offline data to a co-evolving partner.
  • Cross-cutting failure modes: User simulation is especially vulnerable to the sim-to-real gap because prompted assistant LLMs systematically overestimate agent ability.The passage identifies this gap as the most visible recurring failure mode and says it is widest for user simulation.
  • Inference-time roles: Inference-time world models can simulate immediate or multi-step consequences and verify proposed actions before execution.Shallow lookahead, deep tree search, and judge-mediated revision are the depicted roles.
  • Cross-cutting failure modes: Coverage drift occurs when policies enter states outside the simulator’s construction scope, causing simulated responses to diverge from the real environment.This failure follows from deployment beyond the simulator’s covered state distribution.
  • Trends: World models are evolving from static, frozen assets into dynamic partners through a progression from offline trajectories to on-policy rollouts and joint policy co-evolution.This trend is presented alongside a shift toward dedicated user models trained on human dialogue data.

5 Inference-Time World Models

At inference time, text world models guide agents by simulating candidate futures for action selection or by verifying, ranking, and revising policy-generated actions. These approaches range from shallow propose–simulate–score methods to multi-step tree search, but all depend on simulator fidelity.

  • 5 Inference-Time World Models: Inference-time world models support lookahead by predicting consequences of candidate actions before execution, balancing shallow, inexpensive simulation against deeper but costlier search.Methods differ primarily in simulation depth and the resulting compute–quality trade-off.
  • Simulator: Action Selection: One-step methods follow a propose–simulate–score pattern, with WMA using natural-language state differences and value scoring, while WebDreamer selects among simulated outcomes without training.A symbolic variant combines LLM-predicted preconditions and effects with structured rules and Model Predictive Control, achieving the highest reported ALFWorld success.
  • Simulator: Tree Search: Tree-search methods use text world models as transition functions, pairing LLM-generated actions and transitions with MCTS or using one LLM for both modeling and reasoning.Later systems add reflection, offline DPO, executable game rules, or evolving fact memories to improve search and enable self-improvement or online planning.
  • Verifier: Screening and Correction: Verifier-based methods screen policy-generated candidates rather than expanding a search tree, progressing from single-action gates to ranking candidates and regenerating low-confidence actions.Applications include web-task safety, virtual software tests, GUI action ranking, budget-aware tool use, and iterative correction on VisualWebArena.
  • Shared Limitation: Simulator and verifier modes differ in whether they generate futures for selection or screen existing candidates, yet both are bounded by the fidelity of simulator predictions.A simulator is useful when the policy lacks a workable candidate, whereas a verifier presupposes candidates that can be screened or rewritten.

6 Evaluation

Evaluation addresses both the fidelity of text world models and their downstream utility, while also using them as simulators for benchmarking agents. Across these roles, results emphasize that single metrics and simulator outputs can misrepresent competence without long-horizon, interactional, and faithfulness checks.

  • World-model evaluation: World-model evaluation combines next-state prediction accuracy with tests of whether that accuracy translates into downstream task utility.These are identified as complementary evaluation questions.
  • World-model evaluation: Exact-match accuracy is the primary single-step metric, but strong frontier models achieve only modest scores on non-trivial transitions, especially environment-driven and arithmetic-heavy tasks.ByteSized32-State-Prediction introduced this evaluation over many text-game state transitions.
  • World-model evaluation: Multi-step reliability requires consistency measures because sequential prediction errors compound, with consistency remaining high in structured environments but declining in open-ended settings without anchoring.The consistency ratio measures the fraction of world-model-successful trajectories that also succeed in the real environment.
  • Evaluation environments: World models also serve as evaluation environments for dynamic interaction, including environment simulation, user simulation, and domain-specific benchmarks.Examples span GUI state evolution, long-horizon personal dynamics, simulated users, and vertical settings such as earthquake-intensity prediction.
  • Simulator evaluation: 7.4% task-success improvement is reported when a fine-tuned 8B semantic world model supports AndroidWorld, while simulator faithfulness remains essential for trustworthy benchmark conclusions.Faithfulness concerns both realism toward users and the interaction structure, efficiency, and feedback richness of evaluation.

7 Open Problems and Future Directions

Future work should make text world models more deliberate, reusable, and integrated across the agent lifecycle. Key directions include reasoning-capable training, world-model-aware agents, physical grounding, and continual adaptation.

  • Model-policy coupling: World-model design should explicitly choose between shared, adapter-based, and decoupled models, recognizing that isolated accuracy and downstream reward measure different objects.Full sharing is parsimonious when the agent is the only consumer, whereas decoupling supports queryable, benchmarkable, reusable models across policies.
  • Reasoning-capable world models: World models need reasoning-capable prediction methods because direct next-state mapping fails when code execution, scientific simulation, or user modeling requires multi-step inference.Underexplored approaches include CoT distillation, joint reasoning–prediction objectives, and reinforcement learning with process rewards.
  • Unified cross-lifecycle architectures: Unified architectures could serve construction, training, and inference with one model, potentially through multi-task training objectives.Current models are typically designed for a single lifecycle stage, limiting transfer to tree search or evaluation.
  • World-model-aware agent design: Agent architectures should explicitly integrate world-model predictions by deciding when to query them and how much to trust them.Some agents invoke world models in fewer than 1% of episodes, motivating mechanisms such as AVIC’s adaptive gating.
  • Grounded text world models: Grounded text world models should connect digital text predictions to sensor data, physical constraints, and real-world consequences.Systematic grounding in physical observations remains largely unexplored, despite early disaster-assessment and multimodal examples.
  • Continual learning and adaptation: Continual learning must let world models adapt to changing websites, APIs, and user preferences while maintaining accuracy on older environments.Existing test-time adaptation addresses short-term change, whereas long-term continual learning remains unexplored.

8 Conclusion

The survey systematically reviews text world models for LLM-based agents through a formal two-axis framework and the full agent lifecycle, from construction and application to evaluation.

  • Framework: The review organizes text world models by state representation, grounding domain, and lifecycle stage.The lifecycle spans construction, training-time and inference-time application, and evaluation.
  • Construction: It distinguishes LLM-as-WM and Code-as-WM approaches by their assumptions about data, fidelity, and verifiability.These approaches instantiate the transition function under different assumptions.
Loading 2606.09032v1…