Source-linked AI summary
LLM-Only PDDL Domain Repair with Open-Weight Models
Nader Karimi Bavandpour, Pascal Bercher
TL;DR
Constructing reliable PDDL models is difficult, so this paper evaluates whether recent open-weight LLMs can repair them from test constraints. LLMs improve semantic repair quality over a symbolic baseline, but current LLM-only methods do not reliably satisfy every test.
Problem
Constructing planning models remains a practical challenge, motivating evaluation of LLMs for PDDL domain repair.
Method
The paper evaluates recent open-weight LLMs on benchmark PDDL repair using domain-only and test-trace prompts, comparing them with a symbolic optimizer.
Results
The best mean F_1 reaches .87 versus .49 for the symbolic baseline, but the best mean test pass rate is .92 and remains .06 on Thoughtful.
Takeaways & Limitations
Current LLM-only methods improve semantic repair quality but cannot reliably solve the repair problem because every test must pass.
Takeaways & Limitations
Because the benchmark is public and IPC-derived, possible training-data recall limits generality and motivates evaluation on novel unpublished domains.
Abstract
from arXiv · showhide
AI planning is concerned with finding a sequence of actions that achieves a specified goal. It relies on explicit models of the world, commonly represented in the Planning Domain Definition Language (PDDL). An active line of research investigates how errors in such models can be detected and repaired. For example, users may provide positive test plans that are solutions, and negative test plans that fail during execution. Automated repair methods then modify the PDDL model to satisfy these constraints. In this paper, we evaluate the ability of recent open-weight large language models to perform this repair task using an LLM-only approach. Our experiments show that the symbolic baseline achieves an $F_1$ score of $.49$, while the best-performing LLM reaches $.87$ with high reasoning effort, an absolute improvement of $.38$. However, that setting has a mean test pass rate of only $.82$, falling to $.06$ on the Thoughtful domain; even the best setting that includes the test traces reaches only $.92$. Thus, current open-weight models cannot guarantee satisfaction of the test constraints required for reliable automated model repair.
Introduction
The paper frames PDDL domain repair as an interpretable approach to correcting planning models from positive and negative traces. It evaluates recent open-weight LLMs against a symbolic optimizer on a shared benchmark using an LLM-only repair formulation.
- Motivation: Explicit planning models make reasoning over actions, states, and goals transparent, but deploying planning systems remains challenging.
- Domain repair: Domain repair identifies modifications that make positive traces executable and negative traces non-executable.
- Interpretability: Repairs also provide contrastive explanations by addressing why a trace fails and how the domain could have behaved differently.
- Evaluation: The paper investigates recent open-weight LLMs of varying sizes on PDDL domain repair and compares them with a symbolic optimizer.
- Evaluation: All models are evaluated on the benchmark suite introduced by Lin et al. (2025) under an LLM-only repair formulation.
Planning Formalism
A lifted PDDL planning problem separates a domain of predicates and action schemas from a task of objects, initial state, and goal state. Variables are typed and grounded to objects, while applicable grounded actions induce state transitions and solutions are action sequences reaching the goal.
- A lifted planning problem is Π = (P, A, α, O, sI, sg), with domain D = (P, A, α) and task T = (O, sI, sg).
- Objects, Types, and Variables: Variables are typed placeholders for objects, and each type denotes a subset of objects; subtype relations follow subset inclusion.
- Predicates and action schemas are named parameterized structures whose typed variables can be grounded through substitutions that map each variable to a same-type object.
- Actions: An action is applicable when all positive preconditions hold and no negative precondition holds; applying it produces a successor state.
- Solutions: A solution is a grounded action sequence whose state trajectory reaches a state containing the goal facts, with every action applicable in its preceding state.
The Repair Problem
The repair problem formalizes domain modifications as valid sets of atomic operations applied to a planning domain. A solution must preserve positive test plans while making each negative plan fail at its designated first inapplicable action.
- Repair Operations: An atomic repair adds or removes an action schema’s compatible predicate as a positive or negative precondition or effect.It is denoted rJa, p, c, opK, where c identifies the affected component and op specifies addition or removal.
- Repair Operations: A valid repair set contains no pair of atomic repairs that reverses the other’s effect.Applying a valid repair set in any order yields the same modified domain.
- Problem Definition: The domain repair problem is a pair R = (D, T), where each test triple contains a planning problem, finite nonempty positive plans, and finite nonempty negative-plan pairs.Negative plans are undesirable action sequences paired with a position at which failure is expected.
- Solution Conditions: A solution is a valid repair set that transforms D into D′ so every positive plan remains executable and goal-achieving, while each negative plan first fails at its specified action position.The updated planning problem uses D′, and the negative-plan condition identifies the first action that cannot be applied.
Solving the Repair Problem
The section contrasts a sound symbolic repair baseline based on conditional hitting sets with an LLM-only approach that directly predicts repairs. It evaluates trace-free and trace-informed prompting, noting that only the latter directly addresses satisfying test constraints.
- Symbolic baseline: The symbolic baseline uses conditional hitting sets to compute minimal repair sets, applying candidate repairs and iteratively re-evaluating tests until all constraints are satisfied.Ground-truth repairs come from randomly adding or removing preconditions and effects in IPC domains, enabling direct precision and recall evaluation.
- Symbolic baseline: The baseline ignores semantic cues in PDDL names and may miss ground-truth repairs because it optimizes cardinality rather than semantic correctness.When equally sized diagnoses exist, the solver returns an arbitrary one, while the ground-truth repair need not be cardinality-minimal.
- LLM-only approach: The LLM-only approach directly predicts a repair set without symbolic post-processing, extending earlier single-model results to newer LLMs and assessing repair quality.The study examines whether LLM reasoning can usefully complement symbolic repair and whether current models produce sufficiently high-quality repairs.
- Prompt settings: The NOTRACE prompt provides only the corrupted domain, whereas WITHTRACE additionally provides positive and negative test traces and requires repairs satisfying their execution outcomes.Positive traces must be executable and goal-achieving; negative traces must fail at their designated action.
- Prompt settings: WITHTRACE directly addresses the repair problem through test constraints, while NOTRACE addresses a relaxed variant and can propose multiple plausible repairs without verifying the actual instance.LLM outputs may still be noisy or unsound even when test traces are available.
Experiments
Experiments show that recent open-weight LLMs substantially improve repair-set F1 over the symbolic baseline, but their test satisfaction remains insufficient for guaranteed correct repairs. Test traces and higher reasoning effort produce mixed results, with the strongest trace-inclusive setting reaching only .92 mean TR and .06 on Thoughtful.
- Trace inclusion: .85 is GLM 5.2’s highest mean F1 at default effort without test traces; supplying traces lowers its mean F1 to .78.The authors attribute the decrease partly to the combinatorial reasoning burden and possible context-window limits of long plans.
- Reasoning effort: .82 is the best mean F1 with test traces at high effort, shared by DeepSeek V4 Pro and GLM 5.2, below the best default-effort NOTRACE result of .85.Without traces, GLM 5.2 improves from .85 to .87 with higher reasoning effort, while Mistral Small 4 improves from .13 to .41.
- Main results: .87 F1 is achieved by GLM 5.2 with high effort and NOTRACE, versus .49 for the symbolic baseline, while mean TR is only .82.On Thoughtful, this model’s TR falls to .06, satisfying only 6% of tests.
- Test satisfaction: .92 is DeepSeek V4 Pro’s mean TR with WITHTRACE at higher effort, up from .82, while its F1 rises from .74 to .82.Despite this improvement, TR remains below complete test satisfaction and falls to .06 on Thoughtful.
- Implications: The experiments support hybrid methods in which an LLM ranks or filters semantically promising candidates and a symbolic component guarantees test satisfaction.The proposed hybrid approach remains future work.
Conclusion & Future Work
Recent open-weight LLMs identify repairs closer to ground truth than the symbolic baseline, but repair-set overlap does not guarantee test satisfaction. Future work should combine LLM semantic preferences with symbolic guarantees and study prompt, reasoning, and example effects.
- Main messages: The best mean F1 is .87 versus .49 for the symbolic baseline, showing stronger agreement with ground-truth repairs.Reasoning effort strengthens this ability in a model-dependent way, while supplying test traces does not improve the best F1.
- Main messages: The best-F1 setting has a mean TR of .82 but a TR of only .06, so repair-set overlap is not a correctness guarantee.This contrast shows that high repair overlap can coexist with poor test satisfaction.
- Future work: A promising approach combines LLM semantic repair preferences with a symbolic reasoner that preserves theoretical test-satisfaction guarantees.This proposal is motivated by the complementary strengths of the two approaches.
- Future work: Future studies will isolate the effects of prompt length, explicit reasoning requests, and examples.The paper also notes that training-data recall may limit generality because the evaluated IPC-derived benchmark is public.
Appendix: Per-Domain Results
The appendix reports per-domain precision, recall, F1, and test pass rate results for open-weight models across four prompt-and-effort settings. It compares default- and high-effort evaluations with and without test traces, averaging test pass rates across domains.
- Metrics: TR is the fraction of tests satisfied by the repair, and the AVG row averages this rate across domains.For each domain and the AVG row, the best precision, recall, F1, and TR values across models are shown in bold.
- Model groupings: The tables separate models released in 2025 from those released in 2026, with the placement varying by table.Tables 3 and 5 describe upper and lower parts containing models released in 2025 and 2026, while Tables 4 and 6 use corresponding upper/lower sections.