PaperlayerView exact revision on arXiv

Source-linked AI summary

PDDLCoder: Agentic PDDL Generation for LLM-Assisted Symbolic Planning

Veit Laule, Jiangtao Shuai, Manfred Hauswirth, Sonja Schimmler

arXiv:2608.16637v1cs.AI

TL;DR

LLMs often produce invalid long-horizon plans, while existing language-to-PDDL methods use rigid pipelines or incomplete specifications and lack standardized automated evaluation. PDDLCoder iteratively generates, analyzes, and refines PDDL alongside NL-pddlgym, producing applicable plans for 89.6% of tested problems and outperforming prior formalizers and direct LLM planners.

  • Problem

    LLMs often generate long-horizon plans that violate preconditions or fail to reach goals, while existing methods and evaluations remain limited.

  • Method

    PDDLCoder autonomously iteratively generates, analyzes, and refines PDDL using symbolic tools, while NL-pddlgym executes plans to evaluate applicability.

  • Results

    89.6% of tested problems yielded applicable plans, outperforming adapted LLM-Formalizers and direct LLM-Planners on the NL-pddlgym test set.

  • Takeaways & Limitations

    NL-pddlgym provides a 711-problem benchmark with executable environments for reproducible evaluation of LLM-assisted symbolic planning.

  • Takeaways & Limitations

    The approach is studied on fully observable deterministic classical planning and leaves probabilistic domains and other PDDL extensions for future work.

Abstract

from arXiv · show

LLMs remain unreliable for long-horizon planning, often generating logically inconsistent or non-applicable plans. Recent hybrid methods instead translate natural language into the Planning Domain Definition Language (PDDL), allowing symbolic planners to produce verifiable plans. However, existing methods frequently rely on rigid generation pipelines, a partial PDDL definition, or human feedback. Furthermore, their evaluation is hindered by the lack of standardized benchmarks with automated verification. To address these limitations, we present PDDLCoder, an agentic framework for PDDL generation from natural language that iteratively generates, analyzes, and refines planning specifications. We further introduce NL-pddlgym, a benchmark dataset comprising 711 planning problems across 23 domains with executable gym environments for the automated verification of plan applicability. Experiments on the NL-pddlgym test set containing 106 problems across 4 held-out domains show that PDDLCoder generates applicable plans for 89.6\% of tested planning problems. This improves upon our adaptations of previous PDDL generation methods, which achieved up to 45.3\%, and outperforms direct LLM planning approaches, which reached up to 74.5\% on the same test set. Our work demonstrates the effectiveness of agentic PDDL generation for planning and establishes a reproducible benchmark for future research on LLM-assisted symbolic planning.

1 Introduction

The introduction motivates translating natural-language planning tasks into PDDL so symbolic planners can support formal validation, while identifying limitations in existing LLM-based methods. It presents PDDLCoder as an agentic, tool-interacting framework and introduces NL-pddlgym for benchmark-based applicability testing.

  • Motivation: Automated planning requires finding action sequences that transform an initial state into one satisfying a goal while accounting for changing states.Reliable planning therefore depends on accurate environment representations and reasoning about action consequences before execution.
  • Motivation: Direct LLM planners must infer environment dynamics, reason over long horizons, and maintain action validity simultaneously.Such plans can violate preconditions or fail to reach the goal, and lack explicit representations for reliable pre-execution validation.
  • LLM Formalization: LLM formalizers translate natural-language descriptions into PDDL specifications of actions, preconditions, and effects, after which symbolic planners can generate formally guaranteed plans relative to that model.The guarantee depends on the generated model faithfully reflecting the natural-language task.
  • Limitations: Existing LLM formalizers rely on fixed refinement pipelines, partial formal specifications or human feedback, and evaluations that often prioritize syntactic validity over plan verification.These constraints limit autonomous generation and assessment of whether generated plans are executable.
  • Contributions: PDDLCoder autonomously creates, refines, and validates PDDL specifications through external tools, operating zero-shot from natural-language descriptions and action schemas without human intervention or predefined PDDL components.The introduction also presents NL-pddlgym as a benchmark for testing generated plans in matching gym environments.

2 Related Work

Prior work either directly generates plans with LLMs or formalizes natural-language tasks into PDDL, but long-horizon applicability and adaptive, autonomous construction remain challenging. Existing benchmarks evaluate different pipeline stages, motivating NL-pddlgym’s connection between formalization and downstream plan execution.

  • LLM planners: Off-the-shelf LLMs generally struggle to generate applicable long-horizon plans from natural-language input.
  • PDDL formalization: Prior formalization methods generate PDDL from natural-language descriptions, but often rely on predefined domains, decomposition, candidate selection, or fixed refinement schedules.
  • PDDL formalization: PDDLCoder distinguishes syntax errors, unsolvable models, and semantically conflicting plans as separate states that require different tools.
  • Benchmarks: Existing benchmarks cover text-to-PDDL equivalence, domain-model metrics, or downstream planner and validator checks, but leave gaps between formalization and plan-applicability evaluation.
  • Benchmarks: NL-pddlgym supplies high-level task information while withholding predicates, preconditions, effects, and additional parameters, then maps and executes generated plans in target gym environments.

3 Approach

PDDLCoder jointly formalizes natural-language domains and problems as PDDL, then uses LLM agents and symbolic tools to iteratively construct, validate, plan, and refine specifications. NL-pddlgym provides executable environments for evaluating mapped-plan applicability across a held-out benchmark.

  • Problem formulation: PDDLCoder converts natural-language domain and problem descriptions into PDDL files, from which a classical planner derives a plan without human corrections or held-out-environment feedback.The framework targets fully observable, deterministic classical planning problems.
  • NL-pddlgym benchmark: 711 problems across 23 domains form NL-pddlgym, pairing natural-language inputs with executable environments and reserving four complete domains for held-out testing.Final applicability evaluation uses a mapped plan πm matching the environment’s minimal action schemas and object set, generated by MapAgent.
  • Agent architecture: PDDLCoder uses DSPy-orchestrated ReAct to iteratively construct domain and problem files with tools for PDDL creation, reading, editing, and external feedback.Feedback tools include syntax checking, plan generation, and plan-applicability assessment.
  • Symbolic feedback: VAL validates in-memory edits and reports syntax errors, while FD translation provides stricter checking before plan generation.Rejected edits are reported back to the agent for refinement.
  • Planning feedback: The lama-first FD configuration returns the first plan it finds quickly, enabling feedback loops while not guaranteeing plan optimality.Each FD run is capped at 4 GB memory and 1 minute of execution time.

4 Evaluation

PDDLCoder is evaluated on the 106-problem NL-pddlgym test set spanning four held-out domains, using levels that culminate in applicable plans. It reaches 89.6% level-3 success, outperforming adapted prior methods, while ablations and mapping verification show the importance of feedback tools and mapping reliability.

  • Evaluation setup: The test set covers four held-out domains: Elevator, Hanoi, Ring and Peg, and Satellite.Ring and Peg was created specifically for NL-pddlgym, while the other domains are established classical planning domains.
  • Component verification: 98.1% of plans mapped from ground-truth PDDL solved their problems, indicating high reliability for the MapAgent when the initial unmapped plan is correct.The mapped plans were verified by executing them in the appropriate gym environments.
  • Evaluation metric: The evaluation metric defines four increasingly difficult PDDL-generation levels, from syntactic validity at level 0 to applicable problem-solving plans at level 3.A plan is applicable only when every action is applicable in sequence and execution reaches the required outcome.
  • Model comparison: PDDLCoder’s effectiveness depends on LLMs producing usable plan feedback and syntactically valid PDDL, with DeepSeek v4 Flash performing best among the tested models.Gemma 4 31b generated syntactically valid PDDL in 92.5% of cases but frequently failed to identify problems in generated plans.
  • Main results: 89.6% of test problems reached level 3 with PDDLCoder using DeepSeek v4 Flash, compared with 45.3% for NL2Plan, 34.9% for VML_PDDL, 74.5% for COT, and 69.8% for ISR-LLM.The comparison used adapted prior methods on the same NL-pddlgym test set with DeepSeek v4 Flash.
  • Ablation study: Removing feedback tools caused 11% of tested problems to produce syntactically invalid PDDL, showing that external feedback improves syntactic reliability.The ablation retained only CRU tools; the evaluation also included removing plan feedback or plan mapping and a rigid pipeline variant.

5 Conclusion

PDDLCoder is introduced as an autonomous agentic LLM-Formalizer that iteratively creates, analyzes, and refines planning specifications. The work also presents NL-pddlgym, a benchmark of 711 abstract planning problems for evaluating plan applicability.

  • Contributions: PDDLCoder iteratively creates, analyzes, and refines planning specifications as an autonomous, agentic LLM-Formalizer.The paper compares its design to modern coding agents.
  • Contributions: PDDLCoder outperforms adapted LLM-Formalizers and direct LLM-Planners in generating executable, applicable plans on NL-pddlgym.The comparison is reported on the NL-pddlgym test set.
  • Contributions: NL-pddlgym comprises 711 abstract planning problems designed to support rigorous evaluation of plan applicability.The benchmark is presented as part of the paper’s evaluation contribution.

A Experimental Settings

The experiments used a laptop and a four-H100 GPU server, with DeepSeek v4 Flash queried remotely and all other models hosted locally under author-specified sampling settings.

  • Hardware and systems: Experiments ran on a laptop with 64 GB RAM and a GPU server with 100 GB RAM and four NVIDIA H100 GPUs.Both systems used Ubuntu 24.04.4; the laptop had an Intel Core I7-12800H, while the server had an Intel Xeon Platinum 8480C CPU.
  • Model execution: All other models were hosted locally with vLLM using sampling parameters specified by their original authors.The complete parameter settings are listed in Table 6.

B Supplemental Results

The supplemental results report performance levels L0–L3 across models on the NL-pddlgym test set, with tables covering five model configurations and multiple planning methods. ISR-LLM token-consumption statistics are omitted because of a logging limitation, without affecting primary applicability and success-rate metrics.

  • Reporting limitations: L0–L3 performance levels are reported comprehensively for all models, while ISR-LLM token-consumption statistics are omitted.The omission resulted from a logging limitation during local vLLM execution.
  • Supplemental model comparisons: Tables 7–11 report performance across PDDLCoder, VML_PDDL, NL2Plan, ISR-LLM, and COT for five model configurations.The covered models are Gemma 4 31b, gpt-oss-120b, Qwen 3.6 35b A3b, GLM 4.7 Flash, and Llama 4 Scout 17b 16e.

C Example of a Successful PDDLCoder Run

A Ring-and-Peg execution trace shows PDDLCoder detecting an infeasible plan caused by unrestricted ring selection, then refining the domain to enforce single-ring pegs. After 12 iterations, the revised specification produced a logically and physically feasible, applicable plan.

  • Problem setup: The task sorts four colored rings among five same-colored pegs, transferring each ring to its matching peg.The robot arm must move to a peg before picking up a ring.
  • Initial plan diagnosis: The initial 11-action plan is physically infeasible because picking at yellow_peg selects the topmost yellow_ring rather than the intended green_ring.The unrestricted domain cannot select a lower ring from a stack, causing subsequent placements to violate the goal.
  • Domain refinement: PDDLCoder fixes the domain by adding peg_empty, requiring empty destinations for place and marking pegs empty after pick.With five pegs and four rings, the single-ring-per-peg constraint matches the problem and avoids stacking infeasibility.
  • Validated plan: The revised 15-action plan correctly moves the pink, green, and yellow rings to their matching pegs without violating logical or physical constraints.The planning tool generated this plan after both domain and problem files were syntactically correct and solvable.
  • Execution outcome: After 12 iterations, the PlanAgent judged the generated plan properly aligned, and the MapAgent produced a final applicable plan.The final plan was regenerated after external syntactic checks of the domain and problem files.

D PDDLCoder Prompts

The prompts instruct agents to generate complete, solver-ready PDDL while strictly preserving task-specified names and action constraints. Separate prompts support mapping plans into simulation-ready schemas and evaluating their logical and physical feasibility with targeted, actionable feedback.

  • PDDL generation: PDDL-generation prompts require complete, solver-ready domain and problem files that preserve the task’s object, entity, and action names.They prohibit unspecified objects or actions and require a full definition usable by a planner.
  • Plan mapping: MapAgent prompts translate PDDL plans to alternate action schemas and environment object names by matching semantically closest parameters.Unmapped actions remain unchanged, and the same prompt is used for adapted VML_PDDL and NL2Plan plans.
  • Plan evaluation: Feedback prompts assess whether action sequences achieve desired outcomes and remain logically and physically feasible under real-world constraints.They focus exclusively on plan feasibility and applicability rather than unrelated suggestions.
  • Plan evaluation: Critique instructions accept nonoptimal solutions, enforce task-listed actions, and require specific corrective feedback without proposing a replacement plan.Cost annotations and redundant moves are excluded from feedback considerations.

E Rigid PDDLCoder Prompts

The rigid PDDLCoder prompts prescribe staged PDDL generation followed by targeted syntax and solvability repair. They require complete, solver-usable files, strict output formatting, and adherence to the PDDL 1.2 standard where specified.

  • PDDL generation: The generation prompt first requests a complete PDDL domain file whose object and action names follow the natural-language task.The domain must be usable by a solver once paired with the generated problem file.
  • Syntax repair: The syntax-repair prompt supplies erroneous PDDL lines and messages, requiring corrected files as PDDL-only output for direct use in a planning system.It targets syntax errors in either domain or problem files.
  • Syntax repair: The standards-focused repair prompt fixes one file at a time in a domain–problem pair and requires compliance with the PDDL 1.2 standard.The input identifies which file may be changed, while the other remains fixed.
  • Solvability repair: The unsolvability-repair prompt improves only the designated file in an unsolvable pair, requiring syntactically and semantically sound PDDL output.The result is intended for direct use in a planning system.

F COT Prompts · G Adapted Related Work Prompts

The COT prompts constrain plan generation and support mapping PDDL plans across action schemas and object names. The adapted VML_PDDL setup replaces log-likelihood grading with LLM-based semantic scoring for generated domains and problems.

  • F COT Prompts: F COT Prompts require plans to follow the prompt’s object, entity, and action naming.The prompt states that no unlisted objects or actions are allowed.
  • F COT Prompts: F COT Prompts require outputs to be complete and solve the given problem.Correctness is tied to the actions in the plan satisfying their requirements.
  • F COT Prompts: F COT Prompts map original PDDL actions to a different action schema and object naming.The mapping prompt receives the original plan and the target action-schema definition.
  • F COT Prompts: F COT Prompts fit action parameters and object names to semantically closest target-schema parameters.Actions absent from the target schema remain unchanged.
  • F COT Prompts: F COT Prompts use the same MapAgent prompt for COT and final plan mapping in adapted ISR-LLM.This aligns the COT implementation’s mapping prompt with the final mapping step.
  • G Adapted Related Work Prompts: G Adapted Related Work Prompts adapt VML_PDDL to remotely hosted DeepSeek using LLM-based grading instead of log-likelihood grading.Other related-work prompts remain unchanged from their original implementations.
  • G Adapted Related Work Prompts: G Adapted Related Work Prompts score generated domains from 0−100 for semantic alignment with the natural-language description.The domain grader returns only a number.
  • G Adapted Related Work Prompts: G Adapted Related Work Prompts score generated problems from 0−100 for semantic alignment and consistency with the given domain.The problem grader also returns only a number.

H NL-pddlgym Test Set Domain and Problem Descriptions

This section describes planning domains and exemplary problems in NL-pddlgym, including elevator transport, ring stacking, and satellite image capture. The examples specify entities, initial states, actions, and goals for each domain.

  • Elevator: An elevator robot transports passengers from their current floors to desired destination floors.The domain description frames the task as driving an elevator to pick up people and deliver them to requested floors.
  • Elevator: Six floors and three passengers define an elevator instance, with goals to move p0 to f4 and p1 and p2 to f1.The lift starts at f0; passengers begin at f1, f3, and f5, respectively.
  • Hanoi: A ring-stacking robot moves rings between pegs to arrange them largest-to-smallest on one peg.The exemplary instance has 3 pegs and 4 rings initially stacked on peg1, with the goal of stacking them on peg3.
  • Satellite: A satellite-control robot switches instruments and captures images in specified modes using calibrated instruments.Switching on an instrument consumes satellite power until the instrument is switched off.
  • Satellite: Two satellites, four instruments, three image modes, and eight directions define an instance whose goals include multiple captures and satellite0 pointing to Phenomenon5.The goals require infrared0 and spectograph2 images of specified directions, alongside a pointing change for satellite0.
Loading 2608.16637v1…