PaperlayerView exact revision on arXiv

Source-linked AI summary

From Word to World: Can Large Language Models be Implicit Text-based World Models?

Yixia Li, Hongru Wang, Jiahao Qiu, Zhenfei Yin, Dongdong Zhang, Cheng Qian, Zeping Li, Pony Ma, Guanhua Chen, Heng Ji

arXiv:2512.18832v2cs.CL

TL;DR

Agentic RL needs scalable, diverse experience, but real environments are difficult to scale and limited in coverage, motivating world models as simulated interaction sources. This paper evaluates LLMs as text-based world models through next-state prediction across five environments and finds that sufficiently trained models can support coherent long-horizon dynamics and downstream agent gains, while robustness depends on coverage, distributional alignment, and environment complexity.

  • Problem

    It remains unclear whether LLMs can reliably serve as world models and when they meaningfully improve downstream agents.

  • Method

    The paper evaluates LLM-based world models as multi-turn next-state predictors across five text-based environments using fidelity, consistency, scalability, robustness, and agent-utility criteria.

  • Results

    Sufficiently trained LLMs maintain coherent latent state over extended horizons and provide downstream benefits including safer decisions, scalable experience generation, and improved learning efficiency.

  • Takeaways & Limitations

    LLMs can function as implicit text-based world models when trained with dynamics-aligned supervision at sufficient scale and coverage.

  • Takeaways & Limitations

    World-model robustness is constrained by partial observability because the true environment state can be richer than the agent’s textual observations.

Abstract

from arXiv · show

Agentic reinforcement learning increasingly relies on experience-driven scaling, yet real-world environments remain non-adaptive, limited in coverage, and difficult to scale. World models offer a potential way to improve learning efficiency through simulated experience, but it remains unclear whether large language models can reliably serve this role and under what conditions they meaningfully benefit agents. We study these questions in text-based environments, which provide a controlled setting to reinterpret language modeling as next-state prediction under interaction. We introduce a three-level framework for evaluating LLM-based world models: (i) fidelity and consistency, (ii) scalability and robustness, and (iii) agent utility. Across five representative environments, we find that sufficiently trained world models maintain coherent latent state, scale predictably with data and model size, and improve agent performance via action verification, synthetic trajectory generation, and warm-starting reinforcement learning. Meanwhile, these gains depend critically on behavioral coverage and environment complexity, delineating clear boundry on when world modeling effectively supports agent learning.

1. Introduction

Agentic RL faces an experience bottleneck because realistic environments are non-adaptive, difficult to scale, and limited in coverage. This paper studies whether LLMs can act as text-based world models and evaluates their fidelity, scalability, robustness, and utility for agents.

  • Realistic environments constrain experience-driven RL because they are non-adaptive, difficult to scale, and limited in coverage.
  • The paper reframes language modeling in text environments as next-state prediction under interaction, enabling controlled study of LLM-based world models.
  • The evaluation framework covers fidelity and consistency, scalability and robustness, and downstream agent utility.
  • Across five environments, fine-tuned LLMs maintain coherent latent dynamics, scale with model capacity and data, and improve agents through verification, synthetic trajectories, and reinforcement-learning warm starts.
  • The findings support LLMs as text-based world models while identifying behavioral coverage, distribution shift, and environment complexity as important boundaries.

2. Related Works

Prior work often uses environment-specific state abstractions or prompting-based adaptation for LLM world modeling. This paper instead studies free-text multi-turn simulation and evaluates long-horizon consistency and transfer across diverse environments.

  • Existing world-modeling methods commonly predict structured state changes tailored to particular environments and fixed output spaces.
  • The paper formulates world modeling as free-text multi-turn simulation, supporting more general and compositional interaction patterns.
  • It fine-tunes LLMs on large-scale interaction trajectories rather than relying only on zero-shot, few-shot, or lightweight classifier adaptations.
  • Evaluation extends beyond single-step accuracy to rollout stability, WM-to-Real transfer, and generalization across agents, environments, and scales.

3. LLM as Text-based World Models

The paper models text-based interaction as a multi-turn process in which a ReAct-style agent takes natural-language actions and a world model predicts textual state transitions and rewards. It evaluates this formulation across structured and open-ended environments, including partially observable settings.

  • Agent and interaction: A ReAct-style agent alternates internal reasoning with explicit natural-language actions and receives textual environment observations.
  • World Model: The world model predicts next-state transitions and binary rewards from the initial context and interaction history, functioning as an implicit next-state predictor.
  • Partial observability: Because text environments are partially observable, the model may receive fuller initial configurations to approximate latent dynamics beyond the agent’s textual view.
  • Interactive Process: Iteratively combining agent actions with world-model responses produces multi-turn simulated trajectories that capture long-horizon dependencies and cumulative effects.
  • Environments: The benchmark spans bounded, rule-governed environments and open-ended settings with broad compositional dynamics requiring stronger generalization.

4. World Model Training and Evaluation

The study trains and evaluates text-based world models on interaction trajectories from five environments, measuring both one-step fidelity and multi-step rollout consistency. Its setup includes behavioral coverage, partial observability, and transfer-oriented consistency metrics.

  • Data: The data include successful and failed episodes to broaden behavioral coverage, with environment-specific trajectory volumes and a public StableToolBench dataset.
  • Initialization Context: Initialization varies by environment: some models receive full initial states, while partially observable environments require history-based tracking and inference of hidden variables.
  • Training: Training uses multi-turn dialogues of agent actions and environment responses, with supervised fine-tuning predicting the next environment response from dialogue history and the current action.
  • Metrics: The evaluation measures exact-match next-state fidelity and multi-step consistency using Real, WM, W2R, and Consistency Ratio success rates.Consistency Ratio is defined as CR = W2R/Real, with higher values indicating better long-horizon transfer.
  • Evaluation: The benchmark compares prompt-based and fine-tuned models across five environments, using word-level F1 for StableToolBench’s open-ended outputs.

5. Fidelity & Consistency

LLMs show strong next-state fidelity after dynamics-aligned fine-tuning, with consistent long-horizon rollouts in structured environments but drift in open-ended settings.

  • 5.1. Next-state Prediction Fidelity: Pretrained LLMs already support in-context world modeling, including Claude's SciWorld accuracy increase from 56.83 to 73.08 with three examples.Few-shot prompting transfers less effectively to open-ended WebShop, where accuracy plateaus around the mid-50s.
  • 5.1. Next-state Prediction Fidelity: 99%/98% accuracy on ALFWorld and SciWorld, plus 49% F1 on StableToolBench, follows supervised fine-tuning on transition trajectories.These results indicate that dynamics-aligned training captures transition diversity more effectively than prompting alone.
  • 5.2. Rollout Consistency: World models preserve long-horizon consistency at 96%, 91%, and 92% in ALFWorld, SciWorld, and TextWorld, respectively, but typically remain below 80% in WebShop.WebShop's open-ended search diversity makes its dynamics harder to simulate accurately.
  • 5.2. Rollout Consistency: Anchoring rollouts with real search results raises GPT-4o consistency from 56% to nearly 100%, substantially reducing simulation drift.Partial real-environment grounding is especially useful when open-ended dynamics exceed the model's learned coverage.
  • 5.2. Rollout Consistency: Structured environments saturate with roughly 20K training trajectories, while more complex settings benefit from additional data and model capacity.This scaling pattern reflects the greater diversity of open-ended dynamics.
  • 5.2. Rollout Consistency: Consistency depends on agent behavior: GPT-4o-mini is frequently below 70%, whereas GPT-4.1, GPT-5, and Claude reliably exceed 90%.Weaker agents take actions outside the training distribution, motivating broader behavioral coverage in trajectory data.

6. Scalability & Robustness

World-model performance scales with data, model capacity, and environment complexity, while robustness extends across altered layouts, room types, environments, and agent behaviors.

  • Data Scaling: Structured environments improve rapidly and saturate around 20K trajectories, while WebShop benefits up to roughly 70K and StableToolBench shows no saturation at 160K.The contrast reflects low-entropy rule-driven dynamics versus long-tail linguistic and compositional behaviors.
  • Model Size Effects: 1.5B models capture core structured-environment dynamics, but open-ended environments gain steadily from larger models.Open-ended success requires both extensive trajectories and sufficient capacity for long-tailed, high-entropy dynamics.
  • Scalability: World modeling performance scales systematically with data volume, model size, and environment complexity.Structured environments saturate earlier, whereas open-ended environments require more data and capacity.
  • OOD Robustness: World models maintain success rates closely aligned with real-environment performance when ALFWorld layouts change or novel room types appear.The evaluation covers OOD-Seen layout changes and OOD-Unseen room types or configurations.
  • Cross-Environment Transfer: Mixed training consistently accelerates learning and improves final accuracy across progressively broader environment combinations.Mix3 combines ALFWorld, SciWorld, and TextWorld; Mix4 adds WebShop, and Mix5 adds StableToolBench.
  • Behavioral Coverage: Diverse agent behaviors improve OOD consistency, raising GPT-4o-mini’s consistency ratio from 0.49 to 0.81.The result indicates that expert-only trajectories are insufficient under distribution shift.

7. Agent Utility

World models improve downstream agents through pre-execution verification, synthetic trajectory generation, and early exposure to environment dynamics.

  • Pre-Execution Verification: World models simulate irreversible actions before execution, allowing agents to gate checkout on predicted success.WebShop provides the high-stakes setting because checkout ends the episode and errors cannot be undone.
  • Pre-Execution Verification: Moderate verification budgets of 2–10 checks best balance fewer irreversible failures against destabilizing distribution shift.Verification improves success rates, but repeated checks can shift action distributions and weaken imagined-to-real alignment.
  • Synthetic Experience: Synthetic trajectories are evaluated against real-environment trajectories in matched SFT datasets to test whether world-model experience can substitute for interaction.The comparison uses 1,000 successful trajectories under identical training procedures.
  • Early Experience: Early world-model experience consistently improves RL outcomes on ALFWorld and SciWorld.Exposing agents to environment dynamics before policy learning reduces failures driven by incorrect commonsense priors and stabilizes training.
  • Overall Utility: World models provide scalable supervision and early dynamics exposure, reducing reliance on real interaction while improving learning effectiveness.These mechanisms support downstream agent learning through synthetic experience and warm-started policy training.

8. Conclusion

The paper recasts interactive world modeling as multi-turn next-state prediction and evaluates LLMs as implicit text-based world models. It finds useful but conditional benefits for agent learning, governed by coverage, alignment, and complexity.

  • Framework: The study uses text-based environments to evaluate fidelity, scalability, and agent utility under multi-turn next-state prediction.This framework tests whether language models can model interactive worlds rather than only predict sequences.
  • Main Findings: LLMs maintain coherent latent state over extended horizons and provide safer decisions, scalable experience generation, and improved learning efficiency when trained with sufficient scale and coverage.The benefits are reported as tangible downstream improvements rather than as universal properties of all world models.
  • Implication: LLMs can be treated as learned simulators of interactive worlds, bridging next-token prediction with next-state modeling.The conclusion frames this as an empirical foundation for world models in agent learning.

A.1. World Model Training and Evaluation

The study trains and evaluates LLM-based world models across five text-based environments using interaction trajectories, supervised fine-tuning, and controlled agent-learning procedures.

  • Environments: Experiments cover ALFWorld, SciWorld, TextWorld, WebShop, and StableToolBench, spanning different state forms and agent knowledge demands.The environments are summarized along environment nature, required abilities, world state, and world-model capabilities.
  • Data Collection: Interaction trajectories are collected with GPT-4o across four interactive environments, using consistent prompts, protocols, configurations, and up to 50 turns per trajectory.StableToolBench instead uses single-turn API data.
  • Model Training: World models use Qwen2.5-7B and Llama3.1-8B backbones, with Qwen2.5 scale comparisons from 0.5B to 7B parameters.Training uses supervised fine-tuning through LLaMA-Factory.
  • Synthetic Data Evaluation: Matched synthetic-versus-real SFT datasets use 1,000 successful trajectories generated by the same Qwen2.5-7B-Instruct behavior policy.This controls the behavior policy while comparing trajectory source.
  • Agent Learning: Early experience warm-starts agents by predicting the next environment response and termination signal before standard policy SFT and GRPO reinforcement learning.The baseline skips the world-model SFT stage.
  • Initialization Context: Randomized initial states make next-state prediction challenging, so ALFWorld and SciWorld provide initial environment information to the assistant.Object placement, room contents, and house connectivity vary across game instances.

B. Task Examples and Case Studies

Across five environments, the case studies illustrate how agents use text observations, admissible actions, and resulting state changes to complete tasks, while also exposing action-format and exploration challenges.

  • B. Task Examples and Case Studies: The case studies span five environments and show world-model interactions through task instructions, observations, thoughts, actions, and resulting state updates.Examples include household manipulation, tool/API use, temperature-based sorting, navigation, and interactive fiction.
  • B. Task Examples and Case Studies: The household manipulation example completes the task by locating a pillow, moving to the armchair, and transferring the pillow onto it.The final observation confirms that the pillow was moved to the armchair.
  • B. Task Examples and Case Studies: Correct action syntax matters: an invalid “Take pillow 1” produces no change, whereas “take pillow 1 from bed 1” successfully picks up the pillow.The agent revises its action after consulting the available command list.
  • B. Task Examples and Case Studies: The temperature-sorting case study demonstrates iterative exploration: the agent opens doors and moves between rooms before ultimately moving the unknown substance to the yellow box.The intermediate trace includes uncertainty about locating the thermometer and substance.
  • B. Task Examples and Case Studies: The TextWorld example follows a prerequisite chain from opening the antique trunk to retrieving the key and progressing toward the locked wooden door.The trace records the trunk opening, key retrieval, and subsequent door-related actions.

C. Detailed Results

The detailed-results materials document out-of-distribution evaluation in ALFWorld and provide system prompts for trajectory collection across several environments.

  • C. Detailed Results: Table 10 reports ALFWorld task success rates under OOD-Seen and OOD-Unseen settings.OOD-Seen changes layouts within the same room type, whereas OOD-Unseen introduces unseen room types or environment layouts.
  • C. Detailed Results: The supplementary materials include agent system prompts for ALFWorld, WebShop, SciWorld, TextWorld, and StableToolBench trajectory collection.These prompts specify environment-specific interaction formats and collection procedures.
Loading 2512.18832v2…