Source-linked AI summary
Which LLM is Best for Translating Natural Language Goals to PDDL
Tomas Balyo, Lukas Chrpa, G. Michael Youngblood
TL;DR
Formal planning languages such as PDDL can limit access to planning for non-experts, motivating a test of whether LLMs can translate informal video-game goals into PDDL. The evaluation uses an iteratively refined prompt and finds high correctness across models, with meaningful differences in accuracy, speed, and errors.
Problem
PDDL-based planning can be inaccessible to non-experts because expressing objectives requires fluency in formal languages and domain engineering.
Source: Introduction · p. 1Method
The paper evaluates multiple LLMs translating informal video-game testing goals into PDDL, using a prompt template refined through iterative experimentation.
Source: Introduction · p. 1, Prompt Template Design · p. 2Results
96% correctness for Gemini 2.5 Flash, with the fewest false positives (2); all six models exceeded 92% correctness, while GPT-4.1 was fastest and least accurate.
Source: Results · pp. 4–5Takeaways & Limitations
Accuracy, speed, and error differences matter when practitioners choose models for planning pipelines, and failures can reflect language ambiguity or limits in parsing domain logic.
Source: Conclusion · p. 5For inexpressible goals, evaluation was automated, and the quality or detail of the LLMs' natural-language explanations was not manually checked.
Source: Results · p. 4
Abstract
from arXiv · showhide
Bridging the gap between human intent and machine execution remains a challenge in automated planning, where expressing goals in formal languages like PDDL restricts accessibility to non-experts. This paper empirically evaluates whether current Large Language Models (LLMs) can reliably translate natural language testing goals, written in informal language by video game testers, into well-formed PDDL targets suitable for classical planning. We present a carefully designed prompt template, integrating insights from iterative experimentation, aimed at maximizing both accuracy and response coherence from multiple state-of-the-art LLMs. Six contemporary models are systematically assessed on correctness, speed, and error tendencies using real-world, domain-specific benchmarks. All models demonstrate high correctness, exceeding 92\%, with Gemini 2.5 Flash achieving the highest accuracy at 96\% and the lowest incidence of false positives, while GPT-4.1 leads in response speed. Despite these advances, critical distinctions exist in model performance, and occasional failures arise from language ambiguity and limitations in domain representation. Our analysis underscores both the significant progress and ongoing gaps in enabling LLMs to act as robust bridges between natural language objectives and automated planning pipelines.
Introduction
Formal planning languages such as PDDL can make automated planning inaccessible to non-experts. This paper examines LLM translation of informal video-game testing goals into PDDL, evaluating accuracy and speed in a practical setting.
- Formal planning languages such as PDDL require symbolic-logic and domain-engineering fluency, limiting their benefits largely to experts.Source: Introduction · p. 1
- The paper examines whether LLMs can turn plain-language goals into well-formed planning inputs, with the domain model supplied in the prompt.Source: Introduction · p. 1
- Informal video-game testing goals and nuanced planning domains provide a practical setting for evaluating LLM translation accuracy and speed across providers.Source: Introduction · p. 1
Related Work
Prior work studies LLM plan generation and the acquisition of symbolic planning models, while goal specification remains a recurring task for game tests. This paper focuses on translating user goals and providing feedback when a goal cannot be represented in the domain.
- LLMs can generate valid but often limited-complexity robotic plans, while domain-independent planning approaches face scalability limits [Zeng et al. 2023; Wang et al. 2024].Source: Related Work · p. 1
- LLMs have been explored for acquiring symbolic models from natural-language requirements, but domain-model acquisition still requires human experts [Tantakoun, Muise, and Zhu 2025; Smirnov et al. 2024; Vallati et al. 2025].Source: Related Work · p. 2
- Game-testing domain models can be reused across tests, but each test needs a goal; this work also reports when a user goal is inexpressible in the given domain [Xie et al. 2023; Lyu et al. 2023].Source: Related Work · p. 2
Preliminaries
The preliminaries establish the planning and language-model concepts used in the paper. Classical planning represents actions toward goals under defined assumptions, while PDDL organizes a domain and a particular problem instance.
Preliminaries
- The preliminaries introduce automated-planning and LLM concepts needed to understand the paper.Source: Preliminaries · p. 2
Automated Planning
- Classical planning generates action sequences from an initial state toward a goal, assuming a static, deterministic, fully observable, discrete world.Source: Automated Planning · p. 2
- STRIPS represents states with propositions and actions with preconditions and effects; PDDL separates domain rules from a problem instance's objects, initial state, and goal.Source: Automated Planning · p. 2
Large Language Models (LLMs)
- A prompt template is structured input that guides an LLM toward a task or response format, with placeholders for task-specific context.Source: Large Language Models (LLMs) · p. 2
Prompt Template Design
The evaluation uses an iteratively refined prompt template that supplies the domain, typed objects, and natural-language goal. Its revisions aim to reduce long-context errors, false positives, and convoluted outputs.
- Replacing the full problem file with only object definitions followed observed accuracy declines on large problems; the authors hypothesize excessive prompt length contributed.Source: Prompt Template Design · p. 2
- Examples of inexpressible goals were added after earlier prompts elicited false positives, including goals requiring undefined predicates or unavailable actions.Source: Prompt Template Design · pp. 2, 4
- The template supplies the domain file, typed-object list, and natural-language goal to all six evaluated LLMs.Source: Figure 1 · p. 3
- A directive for simple, concise answers was added because convoluted PDDL outputs were often incorrect and difficult for downstream planning.Source: Prompt Template Design · p. 4
Experimental Evaluation
The evaluation tests whether LLMs can map natural-language goals to PDDL expressions or identify goals that cannot be modeled. Across the benchmark, all six models exceeded 92% correctness, with differences in accuracy, speed, and error types.
Experimental Evaluation
- The task supplies a PDDL domain, a problem instance with objects and an initial state but no goal, and an unstructured natural-language goal.The expected output is a valid PDDL goal when expressible, or a brief explanation when it is not.Source: Experimental Evaluation · p. 4
LLM Usage
- The study evaluates two models each from OpenAI, Google, and Anthropic through standard APIs using one prompt per benchmark and the same template.The models include flagship and performance-optimized lightweight or fast versions.Source: LLM Usage · p. 4
Results
- All six models exceeded 92% correctness; Gemini 2.5 Flash reached 96% with two false positives, while GPT-4.1 was fastest.Only one response across all evaluations had invalid syntax.Source: Table 1 · p. 4, Results · pp. 4–5
Benchmarks
- The 90-goal benchmark is balanced between 45 PDDL-expressible and 45 inexpressible goals, spanning four IPC and four game-inspired domains.Goals were crowdsourced from employees, with more than half written by people without prior PDDL experience or access to domain models.Source: Benchmarks · p. 4
Results
- Expressible goals were checked against reference PDDL goals, with expert review when outputs did not match exactly.Inexpressible goals were assessed automatically based on whether the model indicated they could not be modeled; explanation quality was not manually checked.Source: Results · p. 4
- False negatives count missed PDDL expressions for expressible goals, while false positives count generated PDDL goals for inexpressible ones.Figure 2 plots each model’s response time against problem index after sorting prompts by runtime separately for each model.Source: Results · p. 4, Figure 2 · p. 5
Conclusion
The evaluation finds that current LLMs can often translate informal video-game testing goals into PDDL, but performance differences and remaining limitations persist. The authors identify ambiguity and domain-logic edge cases among the challenges and call for broader evaluation and better interactive support.
Conclusion
- The paper’s evaluation of informal video-game testing goals provides a measure of where leading LLMs succeed and remain challenged as PDDL translators.The authors describe the findings as a snapshot and a practical pathway for applying LLMs to automated planning.Source: Conclusion · pp. 5–6
- LLMs routinely achieved high correctness on complex, domain-specific goals, but differed in accuracy, response time, and rates of false positives or syntax errors.The authors identify language ambiguity and limitations in parsing nuanced domain logic or handling cases where the formalism cannot capture human creativity.Source: Conclusion · p. 5
- The authors propose expanding evaluation datasets, refining prompt engineering, and adding feedback loops for dialogue, clarification, and error recovery.These are presented as next steps for handling real-world imperfections.Source: Conclusion · p. 5
Appendix: Natural Language Goals
The appendix directs readers to Tables 2–4 for the natural-language goals used in the experiments.
Appendix: Natural Language Goals
- Tables 2–4 provide the specifications of the natural-language goals used in the experiments.Source: Appendix: Natural Language Goals · p. 6
Natural Language Goals (Not translatable)
The section presents natural-language goal examples from the first-person shooter domain, whose modeled features include weapons, ammunition, and player and bot health.
Natural Language Goals (Not translatable)
- The listed goal examples include eliminating bots, performing headshots, hiding, jumping, using a portal, and checking weapon, ammunition, and health behavior.Source: Natural Language Goals (Not… · p. 7
- The first-person shooter domain models weapons, ammunition, and the health of the player and bots.Source: Table 2 · p. 7
barman
The examples cover IPC parking goals and real-time-strategy goals, with the tables distinguishing translatable goals from goals labeled not translatable.
parking
- Table 3 identifies all IPC-domain goals as translatable; its parking examples place cars at curbs, sometimes with another car directly behind.Source: Table 3 · p. 8, parking · p. 8
Natural Language Goals (Translatable)
- Translatable examples range from producing units and buildings to combat objectives, research levels, and resource collection.Source: Natural Language Goals (Translatable) · p. 9
Natural Language Goals (Not translatable)
- Table 4 lists goals labeled not translatable in a real-time-strategy domain modeling resource gathering, research, and construction of buildings and units.Source: Table 4 · p. 9