Source-linked AI summary
PlanGenLLMs: A Modern Survey of LLM Planning Capabilities
Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan, Fei Liu
TL;DR
LLM planning research lacks consistent criteria and cross-domain comparability because many systems and benchmarks are problem-specific. This survey synthesizes modern LLM planners using six performance criteria, reviews representative systems and evaluations, and identifies future directions and scope boundaries.
Problem
Many LLM planning systems and benchmarks are tailored to specific problems, making cross-domain comparison difficult, while evaluation criteria remain inconsistent.
Method
The survey builds on prior AI planning evaluation work and analyzes representative systems, datasets, methods, and metrics across six criteria.
Results
The survey presents a structured account of LLM planner strengths, weaknesses, evaluation challenges, and research gaps across diverse domains.
Takeaways & Limitations
The proposed criteria provide a foundation for assessing LLM planners and guiding development of robust, adaptable, and efficient systems.
Takeaways & Limitations
The survey primarily covers commonly studied single-agent domains and six criteria, excluding areas such as security and personalization because research remains limited.
Abstract
from arXiv · showhide
LLMs have immense potential for generating plans, transforming an initial world state into a desired goal state. A large body of research has explored the use of LLMs for various planning tasks, from web navigation to travel planning and database querying. However, many of these systems are tailored to specific problems, making it challenging to compare them or determine the best approach for new tasks. There is also a lack of clear and consistent evaluation criteria. Our survey aims to offer a comprehensive overview of current LLM planners to fill this gap. It builds on foundational work by Kartam and Wilkins (1990) and examines six key performance criteria: completeness, executability, optimality, representation, generalization, and efficiency. For each, we provide a thorough analysis of representative works and highlight their strengths and weaknesses. Our paper also identifies crucial future directions, making it a valuable resource for both practitioners and newcomers interested in leveraging LLM planning to support agentic workflows.
1 Introduction
LLM planning research spans diverse methods and tasks, but systems and benchmarks remain difficult to compare across domains. This survey organizes the field around six evaluation criteria and identifies open research needs.
- Many LLM planning systems and benchmarks are tailored to specific problems, limiting cross-domain comparison and selection for new tasks.
- The survey builds on earlier AI planning evaluation work and examines completeness, executability, optimality, representation, generalization, and efficiency.
- It reviews datasets, evaluation methods, and metrics while highlighting gaps involving representation, hallucination, alignment, multi-agent planning, and agentic workflows.
- The taxonomy includes task decomposition, LLM–classical planner hybrids, and search-based approaches among representative planning techniques.
- The survey provides definitions, representative works, and links to appendix tables for readers with different levels of familiarity.
2 LLM Planning Foundations (Tables 1-3)
LLM planning foundations comprise task decomposition, hybrid classical planning, search algorithms, and fine-tuning approaches. These methods differ in how they structure goals, formalize problems, explore alternatives, and adapt model capabilities.
- Task Decomposition: Task decomposition breaks abstract goals into manageable subgoals, enabling intermediate verification and narrower reasoning spaces.
- Task Decomposition: Decomposition may be sequential, parallel, asynchronous, or recursive, with different precondition, effect, and completion requirements.
- LLM + Classical Planner: Hybrid systems use LLMs to translate natural-language problems into formal representations or generate initial plans for classical planners.
- Search Algorithm: Search-based planners systematically explore possibilities using policies, action expansion, world models, and evaluation functions, but can be computationally intensive.
- Fine-tuning: Planning-focused fine-tuning trains models on planning tasks, while generalized agentic fine-tuning combines planning with broader reasoning and tool-use data.
3 Criterion I: Completeness (Table 4)
Completeness requires generating a valid plan when one exists and recognizing when no feasible plan exists. LLMs can produce correct plans with formal solver support but struggle with unsolvable goals.
- Completeness requires both finding a valid plan when one exists and refusing arbitrary plans when no feasible plan exists.
- Correct planning requires achieving the goal within a fixed budget without excessive complexity or infinite loops.
- Classical sound and complete solvers can ensure correctness, provided LLMs accurately translate domains and problems into formats such as PDDL.
- Even GPT-4 and OpenAI O1 struggle to identify inherently unsolvable planning problems because of hallucination issues.
4 Criterion II: Executability (Tables 5-6)
Executability concerns whether plans satisfy environmental action, object, and constraint requirements. Proposed solutions ground objects and actions, refine plans hierarchically, filter candidates, and replan through feedback.
- Executability requires using allowed actions and recognizable objects while satisfying environmental constraints, and it is distinct from correctness.
- Object Grounding: Object grounding restricts plans to objects available in the current environment, such as choosing an available stove instead of an unavailable microwave.
- Action Grounding: Action grounding ensures every planned action is executable, using admissible action lists and decomposition when steps exceed executor capabilities.
- Hierarchical Planning: Hierarchical planning translates high-level steps into executable actions, either by refining all steps afterward or translating them as they are generated.
- Sample-then-Filter: Sample-then-filter methods generate multiple plans and retain candidates that pass correctness and constraint checks.
- Closed-Loop Systems: Closed-loop systems use executor, simulator, validator, LLM, or human feedback to replan until the plan becomes executable.
5 Criterion III: Optimality (Table 7)
Optimality requires reaching the goal through the best possible plan, making it more demanding than standard planning. The survey identifies LLM + Optimizer and A∗ search-based methods as two approaches to optimal planning.
- Optimality means achieving the goal state through the best possible plan, unlike standard planning, which only requires reaching the goal.
- The survey identifies A∗ search-based methods as a second paradigm for achieving optimal plans.
- LLM + Optimizer: LLM + Optimizer systems convert user requests into symbolic optimization problems that an optimizer solves for the best solution.
- LLM + Optimizer: TTG converts minimum-cost travel requests into Mixed Integer Linear Programming problems and uses an optimizer such as SCIP to produce an optimal plan.
6 Criterion IV: Representation (Tab. 8-9)
Representation concerns how LLM-planning inputs and outputs are formatted, with effective formats supporting comprehension and execution. The survey contrasts LLMs as translators into formal languages with standalone LLM planners using structured environmental and plan encodings.
- Representation covers formatted inputs—including domains, problems, and observations—and generated-plan outputs, with effective formats supporting comprehension and execution.
- LLM-as-a-Translator: LLM-as-a-Translator converts between natural language and formal planning languages such as PDDL, making classical planners more accessible to non-experts.
- LLM-as-a-Translator: Natural-language translations into PDDL, LTL, and STL should be tested and checked by validators or human experts for syntax and semantic errors.
- LLM-as-a-Planner: Standalone LLM-as-a-Planner systems encode environments, domains, and plans using tables, symbols, Pythonic code, neural embeddings, and graphs.
- Generalization: Generalization applies learned strategies to more complex out-of-domain scenarios and can use fine-tuning, generalized planning, or skill storage.
- Generalized Planning: Generalized planning extracts common patterns from limited training plans to solve larger, unseen tasks within the same domain.In the Delivery dataset, training on 9–17 locations can generalize to deliveries involving 70–100 locations using the same core strategy.
- Skill Storage: Skill storage learns and reuses previously acquired skills by abstracting them for similar situations and retrieving relevant skills for unseen tasks.
8 Criterion VI: Efficiency (Table 11)
Efficiency targets lower computational and monetary costs by reducing calls, token lengths, interactions, and model sizes. The survey reviews one-shot generation, compact representations, search-based compression, and knowledge distillation as efficiency strategies.
- Efficiency reduces computational and monetary costs by decreasing LLM calls, world-model interactions, input and output lengths, and model sizes.
- Reduced LLM and World Model Calls: Planners reduce LLM and world-model calls by generating whole plans at once, checking feasibility after subgoals, and merging plans with shared prefixes.
- Shorter Inputs and Outputs: Shorter inputs and outputs reduce prompt and plan tokens, while Chain-of-Symbols replaces natural-language descriptions in spatial reasoning trajectories with compact symbols.
- Shorter Inputs and Outputs: Search dynamic bootstrapping iteratively fine-tunes an LLM using shorter solutions while maintaining equal optimality.
- Smaller Model Sizes: Knowledge distillation transfers GPT-3’s planning capabilities into the smaller T5 model to reduce computational burden and costs.
9 Evaluation
The survey evaluates LLM planning with planning-focused and downstream-task datasets, using simulated verification, human assessment, or LLM judges. It also catalogs metrics for criteria such as completeness and plan quality.
- Datasets: LLM-planning datasets are divided into planning-focused datasets and downstream-task datasets.
- Datasets: Planning-focused datasets cover embodied environments, task scheduling, games, and task decomposition, while specialized datasets target grounding, constraints, representation, optimality, and completeness.
- Datasets: Downstream evaluations include agentic tasks such as reasoning, tool use, programming, and web tasks, plus video, image, and text generation.
- Methods: Simulated-environment evaluation validates generated plans with internal or external verifiers, and reference plans can be used when ground truth is available.
- Methods: Human evaluation is used when automated verification is unavailable or when tasks are open-ended and allow multiple valid solutions.
- Methods: LLM-as-a-Judge is faster and more cost-effective than human evaluation for large datasets, but can exhibit position bias, length bias, self-inconsistency, and prompt sensitivity.
- Metrics: Completeness metrics include success rate and goal-condition recall, while true negative rate, false negative rate, and unreachable accuracy assess recognition of unachievable tasks.
10 Discussion
Current LLM planning research faces limitations in evaluation breadth, representation choices, hallucination control, human-preference alignment, token cost, and multi-agent coverage. The survey identifies these constraints and proposes directions for more comprehensive evaluation and improvement.
- Datasets and Baselines: Limited datasets, domains, difficulty levels, criteria, and baselines hinder fair and comprehensive comparisons of LLM planners.The survey recommends standardized leaderboards with consistent metrics, diverse datasets, and both baseline and advanced methods.
- Representation: Natural-language representations dominate despite prompt sensitivity and limited comparisons with PDDL or Python formats.Python representations may improve performance, but translating natural-language problems into Python remains challenging, especially for non-experts.
- Hallucination: Hallucinations can make LLMs misjudge plan achievability or generate inadmissible actions and nonexistent objects.Correcting these outputs may require translation or expert intervention, increasing planning-system cost.
- Human Preference Alignment: Human-preference alignment remains poorly understood, while LLM planners may produce unnecessarily long plans in path planning.The survey notes that humans generally prefer shorter plans for efficiency, simplicity, and cognitive ease.
- Cost Effectiveness: Task decomposition and search-based methods often consume many tokens through lengthy prompts and repeated LLM queries.Suggested improvements include summarizing problem descriptions, improving heuristic evaluation, estimating uncertainty, and verifying plans for early stopping.
11 Conclusion
The survey proposes performance criteria and discusses evaluation challenges to structure assessment of modern LLM planners. It focuses mainly on commonly studied single-agent domains while noting related work in other areas and arguing that similar methodologies support broad coverage.
- 11 Conclusion: The survey proposes key performance criteria and discusses evaluation challenges for modern LLM planners.Its criteria provide a structured approach for assessment across diverse domains.
- 11 Conclusion: Systematic analysis of existing systems, datasets, and evaluation strategies aims to establish a foundation for future LLM-planning research.The survey encourages development of robust, adaptable, and efficient planners.
- 11 Conclusion: The survey primarily covers single-agent robotics, household, and computer-based tasks, while acknowledging applications in natural sciences, IoT, and multi-agent scenarios.It states that similar methodologies and evaluations in these additional areas support the survey’s comprehensiveness.
13 Appendix
The appendix contains summary tables covering foundations, completeness, executability, optimality, representation, generalization, and efficiency in LLM planning.
- Tables 1–3 summarize foundations in LLM planning.
- Tables 4–6 summarize completeness and executability in LLM planning.
- Table 7 summarizes optimality, Tables 8–9 summarize representation, Table 10 summarizes generalization, and Table 11 summarizes efficiency.