Source-linked AI summary
LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models
Marcus Tantakoun, Xiaodan Zhu, Christian Muise
TL;DR
LLMs struggle with reliable long-horizon planning, creating a need to combine their language capabilities with structured Automated Planning. This survey reviews LLM-driven generation and editing of planning specifications and finds that generated PDDL can be parsable and solvable while remaining uncertainly aligned with human goals. It is scoped mainly to PDDL construction and basic deterministic planning support.
Problem
LLMs often produce unreliable long-horizon plans because correctness, optimality, and action effects are not guaranteed at scale.
Method
The survey organizes research on using LLMs to construct and refine AP specifications for domain-independent planners, covering model generation, editing, and benchmarks.
Results
Generated PDDL can be parsable and solvable, but whether specifications align with human goals remains uncertain, particularly in complex domains.
Takeaways & Limitations
Reliable LLM-based planning requires explainable specifications, corrective feedback, external verification, and intermediate representations that support refinement.
Takeaways & Limitations
The survey focuses on PDDL construction and its L2P library currently supports only basic extraction for fully observable deterministic planning.
Abstract
from arXiv · showhide
Large Language Models (LLMs) excel in various natural language tasks but often struggle with long-horizon planning problems requiring structured reasoning. This limitation has drawn interest in integrating neuro-symbolic approaches within the Automated Planning (AP) and Natural Language Processing (NLP) communities. However, identifying optimal AP deployment frameworks can be daunting and introduces new challenges. This paper aims to provide a timely survey of the current research with an in-depth analysis, positioning LLMs as tools for formalizing and refining planning specifications to support reliable off-the-shelf AP planners. By systematically reviewing the current state of research, we highlight methodologies, and identify critical challenges and future directions, hoping to contribute to the joint research on NLP and Automated Planning.
1 Introduction
LLMs struggle with reliable long-horizon planning, motivating a complementary paradigm in which they formalize planning specifications for classical planners. The survey organizes this emerging research around model generation, editing, and benchmarking while identifying challenges and future directions.
- Motivation: LLMs often produce unreliable long-term plans because correctness, optimality, and action effects are not guaranteed at scale.Classical Automated Planning instead uses structured representation, logic, and search methods.
- LLMs-as-Formalizers: LLMs-as-Formalizers construct planning specifications from natural language, while classical planners generate solutions and execute the resulting plans.This paradigm combines LLM capabilities in extracting, interpreting, and refining specifications with automated planning methods.
- Motivation: The survey motivates LLM integration through potential gains in planning accuracy, adaptability to dynamic environments, and agnostic modeling across domains.These motivations include reducing overlooked constraints, manual edits, and reliance on specialized expertise.
- Survey structure: Its taxonomy covers Model Generation, Model Editing, and Model Benchmarks, including task, domain, and hybrid generation within Model Generation.Model Generation translates natural-language inputs into structured planning formalizations, while benchmarks assess planning performance and formalization quality.
- Contribution: The survey presents a comprehensive overview of LLM-driven AP model generation, editing, and benchmarking to support joint research in Automated Planning and NLP.It also summarizes technical approaches, limitations, challenges, opportunities, and future research directions.
2 Background
Automated Planning formalizes action-sequence synthesis over states, actions, and transitions, with PDDL providing a widely used specification language. Within the broader LLMs+AP field, this survey focuses on LLMs-as-Formalizers rather than direct LLM planning or heuristic guidance.
- Automated Planning: Automated Planning synthesizes action sequences that transition an environment from an initial state to a goal state under constraints.Classical planning represents this process using states, initial and goal states, symbolic actions, and a transition function.
- PDDL: PDDL is a widely used formalism for encoding planning tasks as structured specifications that enable off-the-shelf planners to generate robust and optimized solutions.A PDDL model contains domain and problem files defining general rules, constraints, and task-specific information.
- Related work: Recent PDDL-LLM research explores pipelines that balance LLM capabilities and limitations when building neuro-symbolic planning frameworks.Related surveys differ in scope, including broader LLM-augmented planning agents and work beyond traditional Automated Planning.
- Scope: LLMs+AP includes LLMs-as-Heuristics, LLMs-as-Planners, and LLMs-as-Formalizers, but this survey concentrates on the third paradigm.LLMs-as-Formalizers construct AP model specifications that domain-independent planners use to generate solutions.
- Scope: End-to-end LLM planning remains outside the survey’s central focus because it lacks soundness guarantees and may have principled weaknesses.The survey instead organizes formalization work into Model Generation, Model Editing, and Model Benchmarks.
3 LLMs for Constructing Automated Planning Models
This section surveys LLMs for constructing automated planning models, organizing work around model generation, editing, and benchmarks. It highlights intermediate representations, iterative refinement, and external verification as responses to translation ambiguity and model errors, while identifying unresolved semantic, coordination, and evaluation challenges.
- Taxonomy: The survey organizes LLM-driven planning-model research into Model Generation, Model Editing, and Model Benchmarks.Model Generation includes task, domain, and hybrid modeling; the surveyed scope concerns AP specifications such as PDDL.
- Research questions: The central research questions concern aligning planning specifications with human goals and determining how much detail natural-language instructions can accurately convey.These questions frame both goal representation and the translation of natural-language instructions into planning-model definitions.
- Model generation and refinement: Task and domain modeling remain difficult because sparse or ambiguous natural-language descriptions may omit constraints, while a failed single domain model can invalidate planning.The survey points to perceptual grounding, retrieval, commonsense reasoning, and multiple candidate domains as responses to these problems.
- Model generation and refinement: Intermediate representations, consistency checks, reachability analysis, and iterative correction are recurring strategies for reducing one-shot generation errors.Examples include JSON or other structured representations, automated feedback, and dependency analysis before or during conversion to executable planning models.
- Model editing: Current methods show promise for syntactic correction, but semantic errors can still produce non-executable or semantically inconsistent plans.The survey recommends post-hoc analysis using automated metrics or human evaluation as grounded feedback for correction.
- Benchmarks and future directions: Evaluation has advanced, but training-data contamination and cascading pipeline errors complicate reliable assessment and deployment.The survey calls for dynamic benchmarks and notes that linear pipelines can propagate errors across later stages.
4 Language-to-Plan (L2P)
L2P unifies prior natural-language-to-PDDL approaches in an open-source framework that supports extraction, refinement, modular pipeline design, and autonomous end-to-end operation.
- Unified framework: L2P re-implements landmark natural-language-to-PDDL frameworks within a unified, open-source platform.The framework includes reconstructions of prior approaches and supports their implementation and comparison.
- Tool suite: L2P provides a comprehensive tool suite for plugging in different LLMs and running PDDL extraction and refinement experiments.
- Modular design: L2P’s modular design supports flexible PDDL generation, varied prompting styles, and customized pipelines.
- Autonomous capability: L2P supports fully autonomous end-to-end pipelines, reducing reliance on manual PDDL authoring.
- Example reconstruction: The framework includes an action-by-action reconstruction that iteratively generates PDDL actions while updating a dynamic predicate list.
5 Discussion
The discussion finds that LLM-generated planning specifications can be syntactically usable yet semantically misaligned, making verification, feedback, and intermediate representations important for reliability.
- Alignment and verification: Parsable and solvable PDDL files may still fail to align with human goals, especially as domains become more complex.The survey connects this concern to the need for explainable, transparent, and correctable planning outputs.
- Alignment and verification: Corrective feedback loops improve handling of action-precondition errors and unexpected execution failures.
- Refinement mechanisms: External verifier checklists and intermediate representations such as ASP, Python, or JSON are proposed to refine LLM-generated PDDL specifications.
- Prompt sensitivity: Prompting strongly affects translation accuracy and the consistency or executability of generated domain models.Explicit descriptions can improve accuracy, whereas minimal descriptions may leave excessive freedom and produce inconsistent or inexecutable models.
6 Conclusion
The survey reviews nearly 80 works on LLM-based planning-model acquisition, identifies research distributions and gaps, and provides L2P as a foundation for future architectures and more advanced planning languages.
- Survey scope: The survey examines nearly 80 scholarly articles covering frameworks that delegate planning-model acquisition tasks to LLMs.
- Future directions: It generalizes framework methodologies into broader research gaps and future architectural directions.
- Future directions: The L2P library is intended to support applying these methodologies to more advanced planning languages.
Limitations
The survey has two primary limitations: restricted scope and limited technical coverage. It focuses on PDDL construction frameworks, offers brief rather than exhaustive analysis, may omit work from other venues, and currently supports only basic fully observable deterministic planning.
- The survey focuses on PDDL construction frameworks and related papers, leaving other techniques and planning capabilities largely unexplored.The authors note that page constraints also limited each work to a brief overview rather than exhaustive technical analysis.
- Its literature search centers on selected venues, so relevant research published elsewhere may have been missed.
- The L2P library currently supports basic PDDL extraction for fully observable deterministic planning, not areas such as temporal planning.The authors plan to expand the library to broader PDDL applications.
Ethics Statement
The survey identifies no ethical issues beyond those already present in existing LLM-based planning-model construction research. It nevertheless notes potential misuse of L2P to extract actionable planning models for unethical domains.
- The survey reports no ethical issues beyond those already present in existing literature on constructing planning models via LLMs.
- L2P could be misused to extract actionable planning models for unethical domains, a risk the authors describe as pervasive across existing methods.
A Paper Overview
The paper provides a quick summary of core model-generation frameworks and organizes them by planning-specification coverage, publication year, and author. Figure 4 distinguishes explicit PDDL information and feedback or human intervention at the generated specification level.
- The overview summarizes core model-generation frameworks sorted by planning-specification coverage, publication year, and author.
- The framework table records PDDL specification coverage, including initial states, goals, predicates, and actions, alongside prompting, feedback, and human intervention.
- Figure 4 uses Exp. for explicit PDDL information and marks feedback or human intervention supplied at the level of the LLM-generated PDDL specification.
B Additional Information on PDDL
PDDL is a declarative formalism for representing planning domains and tasks, leaving execution to an external planner. Its domain and problem files encode rules, state information, goals, and action dynamics, which classical planners use to generate plans.
- PDDL expresses what should be achieved declaratively, whereas imperative languages such as Python specify step-by-step execution instructions.The distinction is relevant because LLMs can struggle with sequential dependencies in imperative programming.
- A PDDL domain models general planning rules, while a problem file specifies the objects, initial state, and goal state for one task.The Blocksworld example instantiates these two complementary files.
- In Blocksworld, predicates represent properties or relations, and actions define state changes through parameters, preconditions, and effects.For example, pickup requires a clear object on the table and changes the state to holding that object.
- Given the Blocksworld specification, a classical planner can generate a sequence that un-stacks C, places it on the table, and stacks A and B into the goal configuration.
- L2P supports generating domain components and task specifications, combines LLM-generated and human feedback, and includes syntax validation tools.
C.2 Paper Reconstructions
L2P serves as a unified foundation for reconstructing and comparing prior natural-language-to-PDDL frameworks, demonstrating flexibility in standardizing diverse approaches.
- L2P can recreate and encompass previous frameworks for converting natural language to PDDL.
- The library includes multiple paper reconstructions that show how existing methods can be implemented and compared within one system.
- Guan et al.’s 2023 action-by-action algorithm is provided as an example, with its action and predicate outputs shown separately.
C.3 L2P Usage Example
The L2P usage examples illustrate workflows for generating PDDL predicates, task specifications, and formatted predicate-action outputs.
- L2P usage includes generating simple PDDL predicates.
- L2P usage includes generating simple PDDL task specifications.
- L2P usage includes generating LLM feedback on task specifications.
- The examples include predicate and action output generated by gpt-4o-mini through the action-by-action algorithm on the Logistics domain.