PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2World: Benchmarking Large Language Models for Symbolic World Model Generation

Mengkang Hu, Tianxing Chen, Yude Zou, Yuheng Lei, Qiguang Chen, Ming Li, Yao Mu, Hongyuan Zhang, Wenqi Shao, Ping Luo

arXiv:2502.13092v2cs.CLcs.AI

TL;DR

Prior symbolic world-model evaluations faced narrow domain coverage, evaluator randomness, and indirect metrics. Text2World introduces a broad PDDL benchmark with direct, multi-criteria evaluation and benchmarks current LLMs. The results show that reasoning models trained with reinforcement learning perform more strongly, although current models still struggle.

  • Problem

    Prior evaluations of symbolic world-model generation had limited domain scope, relied in some cases on potentially inconsistent LLM-based judgments, and used indirect end-to-end metrics.

    Source: §1 · p. 1

  • Method

    Text2World benchmarks symbolic world-model generation with hundreds of diverse PDDL domains and direct execution-based metrics, including structural similarity and component-wise F1 scores.

    Source: §1 · pp. 1–2, §2.3 · p. 2

  • Results

    Across the benchmark, current LLMs still struggle, while large reasoning models trained with reinforcement learning show stronger world-modeling capabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Increasing test-time budgets consistently improved results, and fine-tuning, in-context learning, and supervised fine-tuning on agent trajectories also contributed positively.

    Source: §1 · p. 2

    Limited online domains prevented construction of a large-scale training set, and manual annotation may introduce subjectivity-related biases.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Recently, there has been growing interest in leveraging large language models (LLMs) to generate symbolic world models from textual descriptions. Although LLMs have been extensively explored in the context of world modeling, prior studies encountered several challenges, including evaluation randomness, dependence on indirect metrics, and a limited domain scope. To address these limitations, we introduce a novel benchmark, Text2World, based on planning domain definition language (PDDL), featuring hundreds of diverse domains and employing multi-criteria, execution-based metrics for a more robust evaluation. We benchmark current LLMs using Text2World and find that reasoning models trained with large-scale reinforcement learning outperform others. However, even the best-performing model still demonstrates limited capabilities in world modeling. Building on these insights, we examine several promising strategies to enhance the world modeling capabilities of LLMs, including test-time scaling, agent training, and more. We hope that Text2World can serve as a crucial resource, laying the groundwork for future research in leveraging LLMs as world models. The project page is available at https://text-to-world.github.io/.

1 Introduction

Text2World addresses limited domain coverage, evaluation randomness, and indirect evaluation in symbolic world-model research with a broad PDDL benchmark and direct, multi-criteria metrics. Benchmark results show that current LLMs still struggle, while reasoning models trained with reinforcement learning perform more strongly.

1 Introduction

  • Prior evaluations were limited by narrow domain coverage, LLM-judge disagreement with humans (Cohen’s κ = 0.10), and indirect end-to-end success metrics.These approaches can constrain generalizability, introduce evaluation error, and make failure modes difficult to identify.Source: §1 · p. 1
  • Text2World is a PDDL-based benchmark built from hundreds of diverse domains to evaluate symbolic world-model generation.The domains were automatically filtered and manually curated.Source: §1 · p. 1
  • Its multi-criteria, execution-based evaluation combines structural similarity and component-wise F1 scores, assessing generated models directly rather than through indirect feedback.Component-wise scores include finer-grained aspects such as action dynamics.Source: §1 · p. 2
  • Text2World’s n-gram contamination rate is lower than prior works (µ = 0.04), supporting evaluation of world modeling rather than pattern memorization.The authors report that complete elimination of contamination remains challenging because PDDL is widely used.Source: §1 · p. 2
  • Across 16 LLMs from 9 model families, the strongest models still struggle, while reinforcement-learning reasoning models perform better and error correction improves performance.Manual analysis found that most errors involved missing essential preconditions or effects.Source: §1 · p. 2
  • Increasing test-time budgets consistently improved results, while fine-tuning, in-context learning, and supervised fine-tuning on agent trajectories also contributed positively.The agent-trajectory result underscores the importance of robust world modeling for developing high-performing agents.Source: §1 · p. 2

2 Preliminary

The paper formalizes symbolic world models and the task of generating them from natural-language descriptions. It evaluates generated PDDL directly using executability, structural similarity, and component-wise scores.

2.1 World Model

  • A symbolic world model consists of fluents, or state variables represented as predicates, and possible actions with parameters, preconditions, and effects.Preconditions specify when an action applies; effects describe how it changes the world state.Source: §2.1 · p. 2

2.2 Task Definition

  • The task maps a natural-language description to a world model that semantically satisfies it, using descriptions of domain objectives, predicates, and actions.Action descriptions intentionally omit explicit preconditions and effects so models must infer dynamics and constraints from descriptive text.Source: §2.2 · p. 2

2.3 Evaluation Metrics

  • Evaluation measures PDDL executability, normalized-Levenshtein structural similarity, and component-wise macro-F1 scores for executable outputs.Component scores cover predicates, parameters, preconditions, and effects.Source: §2.3 · pp. 2–3

3 Benchmark Construction

Text2World was built by collecting, filtering, annotating, and reviewing PDDL domains, then examining dataset characteristics and an LLM-based evaluation approach. The resulting benchmark contains 103 curated domains, and the preliminary human–LLM agreement test found low agreement.

3 Benchmark Construction

  • The benchmark construction process comprises data acquisition, filtering and manual selection, annotation, and quality assurance.The process overview is paired with key statistics for Text2World.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Data Acquisition

  • The acquisition stage collected 1,801 raw PDDL files from public repositories and planning competitions, then standardized their formats.For example, files with BOM encoding were converted to standard UTF-8.Source: §3.1 · p. 3

3.2 Data Filtering and Manual Selection

  • Filtering combined syntax validation, deduplication, complexity and token-length limits, and manual selection, yielding 264 high-quality domain specifications.The pipeline removed near-duplicates above 0.9 cosine similarity, domains exceeding 40 predicates or 20 actions, and files over 5,000 tokens.Source: §3.2 · p. 3

3.3 Data Annotation

  • Six computer-science graduate annotators wrote structured natural-language descriptions emphasizing completeness, inferable dynamics, and natural phrasing without explicit preconditions or effects.The descriptions followed the task format and annotation criteria.Source: §3.3 · p. 3

3.4 Quality Assurance

  • Two senior experts reviewed annotations twice and required both inspectors’ approval; the reported Fleiss Kappa was 0.82, producing 103 curated domains.Contamination analysis found µ = 0.04 versus µ = 0.47 for prior works, though complete elimination remains challenging because PDDL is widely used.Source: §3.4 · p. 4

3.5 Data Analysis

  • The benchmark uses 2 in-context exemplars and 101 test domains; its descriptions commonly concern path planning, constraint satisfaction, and task allocation, and cover eight requirement types.The passage gives :typing as an example of a PDDL requirement.Source: §3.5 · p. 4

3.6 Preliminary Experiment

  • Human annotators and an LLM showed low agreement when judging generated action dynamics (Cohen’s κ = 0.10), motivating more discriminative evaluation metrics.The LLM assessed outputs from claude-3.5-sonnect on Text2World.Source: §3.6 · p. 5

4 Experiments

The experiments evaluate LLM-generated symbolic world models using a zero-shot setup and compare model performance on TEXT2WORLD. The results show that even leading models struggle, while reinforcement-learning-trained reasoning models perform better and correction can improve results.

4.1 Experimental Setup

  • The evaluation included GPT-4, GPT-3.5, Claude-3.5, LLaMA-3.1, DeepSeek-v3, CodeLlaMA, and LLaMA-2, alongside reinforcement-learning-trained reasoning models DeepSeek-R1, OpenAI-o1, and OpenAI-o3.Temperature was set to 0; open-source models ran on NVIDIA A100 GPUs with 80GB memory, while proprietary models were accessed through official APIs.Source: §4.1 · p. 5
  • The experiments use zero-shot prompts with chain-of-thought reasoning, temperature 0, and tarski to check syntactic correctness and executability.For error correction, models refine outputs based on validator-reported syntax errors, with EC3 allowing three attempts.Source: §4.1 · p. 5

4.2 Experimental Results

  • DeepSeek-R1 scores below 60% on both F1PRECOND and F1EFF without error correction, while RL-trained reasoning models outperform others across executability, structural similarity, and component-wise performance.GPT-4 (gpt-4o-mini) executability rises from 48.5% to 72.3% after three correction attempts.Source: §4.2 · p. 5

5 Analysis

The analysis finds a significant performance improvement with three correction attempts and examines how syntax and semantic errors appear and change during correction. It also reports that multiple semantic error types can co-occur.

5.1 Statistical Analysis

  • Three correction attempts significantly improve performance (F = 27.48, p = 0.00012).The analysis excludes anomalous zero values; ECk denotes k correction attempts, with EC0 meaning none and EC3 meaning three.Source: §5.1 · pp. 5–6, Table 1 · p. 6

5.2 Error Analysis

  • UndefinedConstant and IncorrectParentheses errors decrease during correction, while UndefinedDomainName and UndefinedType errors persist.Syntax errors are cases where the generated domain cannot be validated (EXEC. = 0).Source: §5.2 · p. 6
  • Semantic errors include description violations, incomplete models, redundant specifications, and surface variations that preserve semantic equivalence; nearly 10% of cases combine incomplete modeling with redundancy.Different error types can occur simultaneously within a domain.Source: §5.2 · p. 6

6 Exploration

Section 6 explores five strategies for improving LLM world modeling, with reported outcomes varying across models and approaches. The experiments also indicate that more test-time compute and more explicit descriptions can improve performance.

6.1 Test-time Scaling

  • Increasing test-time compute consistently improves model performance for gpt-4o-mini and deepseek-v3.Syntax-parser error feedback is used to assess whether additional compute improves performance.Source: §6.1 · pp. 6–7, Figure 6 · p. 7

6.2 In-Context Learning

  • Few-shot examples produce model-dependent results: claude-3.5-sonnect gains over 20% in component-wise F1, while gpt-4o-mini declines.The demonstrations are structurally similar to, but semantically distinct from, test cases.Source: §6.2 · p. 7

6.3 Fine-tuning

  • Fine-tuning on 601 synthesized PDDL domains brings LLaMA-3.1-70B to performance comparable to GPT-4o-mini, with larger models benefiting more.The domains and descriptions were synthesized with AgentGen.Source: §6.3 · p. 7

6.4 Agent Training

  • LLaMA-2-70B’s world modeling performance improves after training on AgentInstruct, showing a positive correlation with agentic-task performance.The experiment tests whether agent training improves world modeling capabilities.Source: §6.4 · p. 7

6.5 Inference with Concrete Description

  • Concrete descriptions improve performance consistently, supporting the claim that models still struggle to infer action dynamics from abstract descriptions.Unlike abstract descriptions, concrete descriptions explicitly specify preconditions and effects, simplifying the mapping to a world specification.Source: §6.5 · pp. 7–8, Figure 7 · p. 8

7 Related Work

Related work spans neural world modeling and LLM applications, including efforts to generate symbolic domain models. The authors identify a lack of comprehensive benchmarking for this research direction.

7 Related Work

  • Neural world modeling has applications in reinforcement learning, robotics, and autonomous driving, while LLMs have demonstrated zero-shot capabilities across varied tasks.The cited task areas include planning, robotics, and analog design.Source: §7 · p. 8
  • Some LLM world-model approaches predict next states from states and actions, but unreliable outputs can accumulate errors and autoregressive models perform poorly on action effects.Tree-planner instead uses LLMs to construct a possible action space before execution.Source: §7 · p. 8
  • Symbolic-model work includes PDDL generation with human correction, domain synthesis for planning data, and fine-tuning to predict action preconditions and effects.The authors note that this research direction lacks a comprehensive benchmark.Source: §7 · p. 8

8 Conclusion

The conclusion presents TEXT2WORLD as a benchmark for LLM world modeling and reports an evaluation across models. The authors hope it will support future research on using LLMs as world models.

8 Conclusion

  • TEXT2WORLD provides hundreds of domains designed to evaluate LLM world modeling capabilities.The benchmark was developed through a thorough process.Source: §8 · p. 8
  • The authors conducted an extensive evaluation involving 16 LLMs from 9 model families on TEXT2WORLD.Source: §8 · p. 8
  • The authors hope TEXT2WORLD will inspire future research leveraging LLMs as world models.Source: §8 · p. 9

Ethical Considerations

The paper describes data sourcing, annotator recruitment and compensation, bias mitigation, and checks for personal data and ethical compliance.

Ethical Considerations

  • TEXT2WORLD data came from open-source repositories available for academic research, in keeping with the stated commitment to ethical data use.Source: Ethical Considerations · p. 9
  • Graduate-student annotators needed IELTS scores of at least 6; participants gave informed consent and were paid above local minimum wage.Rates were $10 per hour for standard annotators and $20 per hour for senior annotators.Source: Ethical Considerations · p. 9
  • The authors report minimizing geographical-background bias by focusing on human commonsense, and that the dataset contains no personal data.They state that data collection procedures adhere to ethical guidelines.Source: Ethical Considerations · p. 9

Limitation

The paper notes that limited online domain availability prevented construction of a large-scale training set, and illustrates its PDDL domain format with a robot navigating a grid of keys and locked doors.

Limitation

  • Limited online domain availability prevented construction of a large-scale training set; future work should add sources such as synthesized data, while manual annotation may still introduce bias.The authors note that human subjectivity cannot be fully eliminated despite regular inspections.Source: Limitation · p. 9

A.1.1 Domain Example

  • The domain example represents robot movement, key handling, and matching keys to locks through PDDL predicates and actions.The example describes a grid with places, keys, and doors whose locks have corresponding shapes.Source: §A.1.1 · p. 12, §A.1.2 · p. 12

A.1.2 Abstract Description

  • The abstract description lists grid places, keys, and locks alongside actions for unlocking, moving, picking up, swapping, and putting down keys.Unlocking uses a key of a specific shape.Source: §A.1.2 · pp. 12–13, §A.1.3 · p. 14

A.1.3 Concrete Description

  • The concrete description specifies action conditions and effects, including requiring a matching held key to unlock a connected locked destination.Moving requires a connection and an open destination.Source: §A.1.3 · p. 14

A.2 Preliminary Experiment

The preliminary experiment evaluates LLMs on PDDL error correction, focusing on semantic-error detection and zero- and few-shot prompting, and describes dataset requirements and evaluation metrics. It reports weak semantic-error detection and presents the syntax- and semantic-error distribution for claude-3.5-sonnect under few-shot prompting.

A.2 Preliminary Experiment

  • Semantic-error detection achieved 55.0% accuracy, 56.2% precision, 45.0% recall, 50.0% F1, and 55.0 ROC AUC, near random performance.The authors conclude that this makes reliable distinction between correct and incorrect PDDL domains difficult.Source: §A.2 · p. 14

A.3 More Details on Data Analysis

B.1 Evaluation Metrics

  • Evaluation uses Levenshtein Ratio for string similarity and component-wise F1 scores for predicate and action similarity.The passage describes Levenshtein Ratio as ranging from 0 to 1 and component-wise F1 as the harmonic mean of precision and recall.Source: §B.1 · p. 15

B.2.1 Error Correction

  • The correction prompt requires complete corrected PDDL and checks domain constructs, action definitions, parameter counts, typed objects, and the use of '-' in :types.It notes that typed domains should list their object types and that '-' should not appear in :types.Source: §B.2.1 · p. 16

B.2.2 Zero-Shot Prompt

  • The zero-shot prompt asks the model to convert a domain description into a well-structured PDDL domain adhering to syntax and semantics, with step-by-step reasoning.Source: §B.2.2 · pp. 17–18

B.2.3 Few-Shot Prompt

  • The few-shot prompt supplies gripper and blocks-world PDDL examples before asking the model to generate a domain from a new description.Source: §B.2.3 · pp. 18–20
  • The few-shot prompt describes robot movement to adjacent tiles, including tiles directly below, to the right, and to the left of the robot’s current position.Source: §B.2.3 · pp. 20–21

C.1 Overall

  • Table 3 presents the distribution of syntax and semantic errors for claude-3.5-sonnect under the few-shot setting.The supplied passages do not report the table’s individual values.Source: §C.1 · p. 21, Table 3 · p. 21

C.2 Syntax Error

The appendix identifies tables for syntax-error and semantic-error distributions, followed by a comparison of LLM performance on concrete domain descriptions.

C.2 Syntax Error

  • Table 4 presents the distribution of syntax errors in PDDL generation across 66 samples.A task may have one to four samples.Source: Table 4 · p. 21, §C.2 · p. 21

C.3 Semantic Error

  • Table 5 presents the distribution of semantic errors in PDDL generation across 91 samples.A task may have multiple semantic errors.Source: Table 5 · p. 21, §C.3 · p. 21

D.1 Experimental Results with Concrete Description

  • Table 6 compares LLM performance on TEXT2WORLD using concrete domain descriptions, including settings with zero or three correction attempts.EC0 is zero-shot without correction; EC3 allows three correction attempts.Source: Table 6 · p. 22
Loading 2502.13092v2…