PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

Joana Rosa, Pedro Santos, Valdemar Oliveira, Romão Silva, L. Miguel Silveira, Bruno Martins

arXiv:2609.09898v1cs.AI

TL;DR

NL-to-PDDL systems can produce parseable and solvable problems that still misrepresent the intended task. This paper combines generation, operational checks, criticism, and iterative repair, then compares accepted outputs with curated references offline. Operational success and reference reconstruction diverge substantially, while repair helps in some settings and PDDL 2.1 remains difficult.

  • Problem

    Parseability and planner success do not reliably establish that an NL-to-PDDL problem preserves the intended objects, state, goals, metrics, or semantics.

  • Method

    The pipeline combines LLM generation with parsing, domain-conformance, planning, VAL validation, LLM criticism, iterative repair, and offline structural or semantic reference comparison.

  • Results

    Operational success and benchmark-reference reconstruction diverge: Planetarium+FS reaches 0.583 operational acceptance after repair but only 0.417 final semantic and structural reference matching.

  • Takeaways & Limitations

    Operational acceptance should be used for deployment-oriented stopping, while controlled benchmarks should separately measure reconstruction of curated reference instances.

  • Takeaways & Limitations

    The online criterion is only a proxy for task faithfulness, and the pipeline assumes a correct domain.pddl rather than generating domain models.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standard evaluation criteria such as syntactic validity or planner success can substantially overestimate faithfulness to the described task: a generated problem may be parseable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target. This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, an LLM critic, and iterative repair. Fine-grained repair feedback is constructed from the domain description, the generated problem, the natural language problem description, and operational diagnostics. Reference-based comparisons against curated benchmark PDDL problem descriptions are used for post-hoc benchmark analysis, and these offline checks include renaming-invariant structural matching and semantic equivalence, where domain support is available. Across Planetarium, AutoPlanBench, and curated PDDL~2.1 problems, results show that operational success and benchmark-reference reconstruction can diverge substantially. Results also show that structured repair can be useful, and that PDDL~2.1 remains challenging for reference reconstruction, even when operational success improves.

1 Introduction

NL-to-PDDL generation can make symbolic planning accessible, but evaluating generated problems requires more than parseability or planner success. The paper studies an end-to-end generation, evaluation, and repair pipeline against operational and reference-based criteria.

  • Small errors in objects, initial predicates, goals, optimization metrics, or numeric fluents can significantly alter the resulting planning problem.
  • Parseable and solvable instances may still encode the wrong initial state, omit predicates, specify an incorrect goal, or optimize the wrong quantity.
  • The study evaluates deployment-realistic generation with few-shot examples and iterative repairs, compares operational acceptance with benchmark reconstruction, and examines transfer to PDDL 2.1.
  • Few-shot prompting and iterative repair improve operational performance in several settings, while reconstructing curated references remains difficult, especially for PDDL 2.1.
  • The pipeline combines parser, domain-conformance, planner, VAL validation, and natural-language critic feedback in an iterative problem-generation loop.

2 Background

Related work separates language models used as direct planners from models that formalize natural-language descriptions for symbolic planning. Feedback, validators, and formal reasoning tools increasingly support more reliable planning-model generation.

  • One research direction uses language models to generate or select plans directly, while another translates user descriptions into structured representations for symbolic planners.
  • Planning formalization ranges from translating restricted natural-language goals to recovering complete planning representations from richer text.
  • Recent datasets show that performance degrades when inputs become less templated and more semantically demanding.
  • Prior systems refine generated planning models using PDDL validators, human corrective feedback, environment interaction, and symbolic feedback-driven search.
  • Formal verification and satisfiability-based reasoning have also been combined with language models to address planning problems more reliably than direct plan generation alone.

3 Method

The method generates complete problem.pddl instances from natural-language descriptions and a domain file, then evaluates and repairs them through structured operational feedback. Offline structural and semantic comparisons assess reconstruction of benchmark references when available.

  • The pipeline repeatedly generates, checks, and either accepts or repairs a candidate problem.pddl using the natural-language description and PDDL domain.
  • Generated instances include objects, initial state, goals, and applicable metrics or numeric initializations consistent with the domain vocabulary and action schemas.
  • Few-shot examples demonstrate how same-domain descriptions map to object declarations, initial facts, and goal conditions.
  • Operational evaluation checks parsing, local domain conformance, planning, VAL validation, and LLM-critic acceptance before accepting a candidate.
  • Repair feedback combines coarse outcomes with fine-grained checker, planner, validator, and critic diagnostics, and the loop stops on success, stagnation, unavailable guidance, or attempt exhaustion.
  • Semantic and structural reference comparisons are computed post hoc because curated reference problems are unavailable inside the online repair loop.
  • Offline structural matching compares objects, initial facts, goals, metrics, and PDDL 2.1 numeric values while allowing consistent object renaming.
  • The same pipeline spans Planetarium and AutoPlanBench, using semantic equivalence where supported and structural comparison elsewhere.

4 Experimental Setup

Experiments cover classical and PDDL 2.1 benchmarks under baseline and few-shot prompting, with up to three generation or repair attempts. Evaluation separates operational acceptance from offline reference reconstruction using structural and, where supported, semantic metrics.

  • The experiments use gpt-4.1-mini for generation and criticism, with one initial generation and up to two repair iterations at temperature 0.0.
  • Baseline prompting uses the domain and natural-language description, while the few-shot condition adds one same-domain example selected by dataset-specific procedures.
  • The repair prompt supplies the previous draft and structured evaluator feedback, while the critic separately assesses the description, domain, generated problem, and diagnostics.
  • Planning checks use Fast Downward and VAL for classical domains, TFD and VAL for time-simple PDDL 2.1 domains, and ENHSP and VAL for numeric-fluent domains.
  • A candidate is operationally accepted only when it parses, conforms to the domain, yields a plan, passes VAL, and satisfies the critic.
  • Reference analysis uses renaming-invariant structural matching across objects, initial state, goals, and metrics, with semantic equivalence where domain support exists.
  • Runs evaluate 60 examples unless stated otherwise, report component rates and runtime, and distinguish final results from Step-0 gains.
  • Repair gain is defined as the final reference-match rate minus the Step-0 reference-match rate.

5 Experimental Results

Across classical and PDDL 2.1 benchmarks, few-shot prompting and iterative repair improved some operational outcomes, but operational acceptance often diverged from reference reconstruction, especially for temporal and numeric problems.

  • Supported classical domains: Few-shot prompting raised Planetarium semantic reference matching from 0.267 to 0.417 and structural matching from 0.200 to 0.417.
  • Supported classical domains: On supported AutoPlanBench, few-shot prompting lowered final semantic and structural matches from 0.842 to 0.714.
  • Operational outcomes: Planetarium final operational success increased from 0.500 to 0.583 with few-shot prompting, while APB decreased from 0.772 to 0.750.
  • Unsupported classical domains: On unsupported APB, few-shot prompting raised structural exact match from 0.567 to 0.650, although structural gain was negative in both conditions.
  • PDDL 2.1: PDDL 2.1 structural exact match remained low, rising from 0.000 to 0.067 despite operational improvements.Few-shot prompting also raised final operational success from 0.500 to 0.733, with solve and VAL-valid rates rising from 0.550 to 0.750.

6 Discussion

The discussion finds that operational success, reference reconstruction, and few-shot benefits are distinct and benchmark-dependent. The framework transfers across settings, but PDDL 2.1 remains much harder to reconstruct exactly than to execute operationally.

  • Operational acceptance is not interchangeable with semantic or structural reference reconstruction, so both signals matter for evaluation.
  • Few-shot prompting helps unevenly: it improves Planetarium, unsupported APB, and PDDL 2.1, but lowers supported-APB operational and reference-based metrics.
  • In PDDL 2.1, operational success is easier to obtain than exact structural agreement, indicating a need for stronger feedback on numeric initializations, metric clauses, and reference object structure.
  • The main error patterns include incomplete initial states, reference mismatches, representation-sensitive object, predicate, and metric mismatches, and omitted metrics or incomplete numeric initializations in PDDL 2.1.

7 Limitations and Future Work

The study’s scope is limited by proxy-based faithfulness checks, relatively formal language, an assumed correct domain file, uncertain critic reliability, single-turn interaction, and limited PDDL 2.1 coverage.

  • Evaluation scope: Operational acceptance is only a proxy for task faithfulness because the checker, planner, validator, and LLM critic can miss mismatches.The pipeline does not yet compare behavior induced by generated and reference instances.
  • Language scope: The benchmark descriptions are relatively formal and do not fully represent open-ended language involving informality, underspecification, or common-sense inference.The authors identify broader linguistic evaluation as a natural next step.
  • Task scope: The pipeline assumes a correct domain.pddl file and generates only problem.pddl descriptions, leaving domain-model generation outside its scope.Extending the framework to generate domains would broaden the setup beyond controlled problem benchmarking.
  • Critic reliability: The LLM critic agreed with manual faithfulness judgments in 13/24 audited cases, with both false positives and false negatives.Larger studies are needed to assess reliability across domains, prompts, and error types.
  • Interaction scope: The study uses single-turn interaction, leaving clarification-driven revision unexplored for informal, incomplete, or ambiguous descriptions.The authors propose explaining generated PDDL back to users and revising it through dialogue.
  • PDDL 2.1 scope: The PDDL 2.1 extension covers six curated variants and does not establish performance on richer temporal, numeric, or broader planner settings.Its low structural exact-match rate indicates that operational checks and critic feedback do not reliably recover metric-sensitive instances.

8 Conclusions

The paper concludes that NL-to-PDDL evaluation must distinguish operational acceptance from faithful reconstruction of curated reference instances. Few-shot prompting and repair help unevenly across settings, while stronger critics, benchmark analysis, and cross-setting repair remain necessary.

  • Conclusion: Parseability and planner success alone cannot reliably establish that an NL-to-PDDL instance faithfully represents the intended task.The pipeline therefore combines operational checks with offline semantic-equivalence and renaming-invariant structural comparisons.
  • Repair and prompting: Few-shot prompting improves operational success in Planetarium, unsupported AutoPlanBench, and PDDL 2.1, but not supported AutoPlanBench.Repair is most visible when operational feedback converts an executable near-miss into an accepted instance.
  • Implications: Operational repair should use operational evidence, while controlled benchmarks should separately measure reconstruction of curated reference instances.The authors identify stronger generators, critics, benchmark analysis, and cross-setting repair as complementary needs.

A Additional Error Pattern Discussion

Errors recur in incomplete initial states, plausible but incorrect reference encodings, representation-sensitive fields, and PDDL 2.1 metric or numeric details.

  • Error categories: Incomplete or inaccurate initial states can prevent structural recovery or semantic equivalence through omitted or incorrect predicates.Small initialization errors are sufficient to break stricter reference matching.
  • Error categories: Some outputs are parseable, solvable, and VAL-valid yet still fail offline semantic or structural reference matching.Operational validity does not guarantee reconstruction of the curated benchmark encoding.
  • Error categories: Representation-sensitive mismatches affect object declarations, initialization facts, predicates, or metrics despite otherwise plausible task structure.These errors preserve much of the apparent task while differing from the benchmark representation.
  • Error categories: PDDL 2.1 adds metric-specific failures, including omitted or incorrect :metric clauses and incomplete numeric initializations.The appendix situates these patterns alongside the paper’s classical and unsupported AutoPlanBench diagnostics.

C Results with Different LLMs

The paper compares operational evaluation and repair across model settings, while distinguishing operational acceptance from semantic and structural benchmark fidelity. Stronger models approach operational saturation on classical tasks, but benchmark fidelity remains variable, especially for PDDL 2.1.

  • PDDL 2.1: The PDDL 2.1 experiments evaluate operational component checks and repair outcomes across six curated domain variants.The corresponding comparisons report operational acceptance against structural benchmark fidelity.
  • Model comparison: Across gpt-5.5 and opus-4.7, operational success is near-saturated on most classical settings, while benchmark fidelity remains more variable.The clearest residual gap is in PDDL 2.1, where structural recovery stays low despite high operational acceptance.
  • Evaluation setup: The experiments report operational component checks and repair outcomes separately from semantic and structural benchmark-fidelity comparisons.The latter are offline benchmark metrics, whereas operational columns use the operational stopping criterion.
  • Generation and repair: The pipeline reuses one generation step throughout iterative repair, accepting or revising candidate problems using structured feedback.The same architecture supports classical PDDL and PDDL 2.1, with metric-sensitive checking and plan-related requirements for the latter.
  • Critic behavior: The critic prompt instructs acceptance when formal diagnostics succeed unless a concrete natural-language contradiction or omission is identified.It also emphasizes cardinality, predicate signatures, domain executability, and stable minimal repair guidance.

F Manual Audit Examples

The manual audit illustrates two ways operational checks and critic judgments can diverge from task faithfulness: omitted optimization objectives and unjustified rejection of valid domain bookkeeping.

  • Accepted but unfaithful: A critic accepted an executable rover instance that captured the sampling goal but omitted the requested metric minimizing recharges.The audit classifies this as a false positive because operational checks succeeded while a metric-sensitive requirement was lost.
  • Rejected but faithful: A depot instance was rejected even though manual inspection found it faithful to the natural-language description and curated reference encoding.The example included crate 0 being clear while resting on crate 2.
  • Critic errors: The critic’s rejection was a false negative caused by treating clear and on as contradictory, although the depot encoding permits a clear crate to rest on another crate.Here, clear means that no object is on top of the crate.
Loading 2609.09898v1…