PaperlayerView exact revision on arXiv

Source-linked AI summary

From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning

Chenghao Zhang, Yikai Mao, Shanqi Liu, Haoyu Gao, SaiSai Hu, Dan Roth

arXiv:2608.21897v1cs.AI

TL;DR

Reliable natural-language-to-PDDL planning is difficult because solver success can diverge from semantic faithfulness, while human-written PDDL demonstrations are costly. The paper addresses this with a solver-grounded reinforcement-learning framework using shared Actor, Judge, and Editor roles, and reports broader planning gains with improved faithfulness.

  • Problem

    Reliable planning needs faithful natural-language-to-PDDL formalization, but solver success alone can reward specifications that diverge from the original task intent.

  • Method

    A single language model jointly performs specification generation, solver-calibrated verification, and bounded diagnostic-conditioned repair using deterministic solver feedback without human annotations.

  • Results

    Across PlanBench domains, the method improves planning success and faithful success while substantially reducing specification drift relative to solver-only learning.

  • Takeaways & Limitations

    Organizing solver feedback into generation, verification, and repair supports more scalable and faithful annotation-free symbolic planning.

Abstract

from arXiv · show

Reliable planning requires converting natural-language instructions into executable symbolic specifications, yet large language models remain brittle without costly PDDL annotations and may exploit solver success in semantically unfaithful ways. We study how to learn faithful natural-language-to-PDDL formalization using only solver feedback, without human-written demonstrations. We propose a solvergrounded multi-role reinforcement learning framework where a single language model acts as an Actor, Judge, and Editor for generation, verification, and repair. The Actor proposes PDDL specifications, the Judge provides a solver-calibrated quality signal, and the Editor performs bounded diagnostic-conditioned refinement. On PlanBench, our method improves average success from 35.5% for LLM+P to 70.8%, achieves 66.3% faithful success, and reduces semantic drift to 6.4%. These results show that organizing solver feedback into generation, verification, and repair roles enables more scalable and faithful annotation-free symbolic planning

1 Introduction

Reliable symbolic planning requires faithful translation from natural language into executable formal specifications, but solver success alone can reward semantically shifted specifications. The paper proposes solver-grounded multi-role learning without human PDDL demonstrations.

  • 1 Introduction: Structured planning transforms high-level instructions into executable action models under explicit state-transition constraints.The paper motivates applications including robotics, workflow automation, logistics, and interactive decision-making.
  • 1 Introduction: LLMs remain brittle on planning because reliable solutions require logical consistency, long-horizon state tracking, and faithful task-constraint grounding.
  • 1 Introduction: Solver success is necessary but insufficient because generated specifications can be weakened or semantically shifted while remaining easy to execute.
  • 1 Introduction: The paper asks whether an LLM can learn reliable symbolic planning from solver feedback alone, without human-annotated PDDL demonstrations.
  • 1 Introduction: Solver-Grounded Multi-Role Reinforcement Learning assigns one LLM coordinated Actor, Judge, and Editor roles for generation, verification, and diagnostic-conditioned repair.

2 Related Work

Prior work exposes gaps between fluent language reasoning and executable planning, motivates LLMs as formalizers, and explores feedback-driven self-improvement under verifiable rewards.

  • LLMs as planners: Planning benchmarks show that fluent rationales do not guarantee valid state-transition reasoning, especially under predicate obfuscation and longer horizons.
  • LLMs as planners: NATURAL PLAN reports sharply worse realistic-language planning as constraint complexity increases.
  • LLM-as-formalizer and neuro-symbolic planning: LLM+P delegates search to a classical planner by translating natural language into PDDL and verbalizing the resulting plan.
  • LLM-as-formalizer and neuro-symbolic planning: LLM-as-formalizer approaches face supervision bottlenecks involving curated demonstrations, fixed domain assumptions, or supervised initialization.
  • Feedback-driven self-improvement and verifiable rewards: Feedback-driven systems use critique–revise or environment-feedback loops, while judge-based methods can risk evaluator bias and reward hacking without external grounding.

3 Method

The method trains one shared language model in Actor, Judge, and Editor roles around deterministic solver feedback. Failed specifications receive bounded, diagnostic-conditioned repairs, while role-specific learning signals optimize generation, calibration, and repair.

  • 3.1 Task and Solver-Grounded Environment: Each task maps a natural-language description to a PDDL domain and problem specification evaluated by a deterministic solver.The solver returns a binary executability label and structured diagnostics.
  • 3.1 Task and Solver-Grounded Environment: Raw solver optimization can reward degenerate specifications that weaken goals, remove constraints, or alter predicates.
  • 3.2 Role-Conditioned Multi-Role Policy: A shared language-model backbone is conditioned into Actor, Judge, and Editor roles with role-specific contexts and lightweight parameters.The shared backbone contains approximately 95% of the parameters.
  • 3.3 Solver-Grounded Training Episode: The Actor samples an initial specification, the solver returns success and diagnostics, and the Editor performs at most Hmax repairs when execution fails.The episode stops after solver success or when the repair budget is exhausted.
  • 3.4 Role-Specific Learning Signals: The Actor uses solver and Judge signals, the Judge predicts solver success, and the Editor is rewarded for successful repairs with fewer edits.Actor and Editor policies use PPO, while the Judge is trained as a calibrated binary predictor.

Appendix B.2 specifies the PPO objective, banded Judge penalty, decoding policy, and hyperparameters.

The framework links solver outcomes to Judge calibration and shared-role learning, while parameter sharing restricts local reward-hacking directions without eliminating them.

  • Judge calibration: The Judge is trained on solver outcomes, so its shaping signal is reliable when solver success and task-level correctness disagree only rarely.The paper formalizes this connection through on-policy disagreement and Judge calibration error.
  • Multi-role design: The Actor generates global formalizations, the Judge converts sparse solver outcomes into calibrated scores, and the Editor performs diagnostic-conditioned local repairs.The three roles are coordinated within one model rather than optimized solely through a final outcome.
  • Parameter sharing: The shared-backbone design restricts Judge- and Editor-neutral Actor perturbations to a smaller role-private subspace than fully separate models.Most shared-backbone directions are observed by the Judge and Editor objectives.
  • Reward-hacking analysis: Parameter sharing reduces local degrees of freedom for improving Actor reward without corresponding verification and repair changes, but does not eliminate reward hacking.The comparison is framed as a local first-order analysis around a reference parameter point.

4 Experiments

Experiments evaluate the method across PlanBench, zero-shot transfer benchmarks, matched solver budgets, faithfulness metrics, repair dynamics, Judge separation, and role ablations. Results show gains in planning success alongside improved semantic faithfulness and reduced drift.

  • Overall performance: 70.8% average PlanBench success exceeds 35.5% for LLM+P, with the method achieving the best performance across all four evaluated PlanBench domains.The evaluation also includes zero-shot transfer to ProntoQA and NATURAL PLAN.
  • Controlled comparison: 21.0 points of average PlanBench success separate the method from the strongest same-backbone baseline under a matched K = 6 solver-call budget.The method also has the highest faithful success and lowest semantic drift in this controlled comparison.
  • Semantic faithfulness: 66.3% faithful success accompanies 70.8% solvability, while semantic drift is reduced to 6.4%.Faithful success requires solver executability and semantic consistency with the reference task.
  • Ablation and drift: Removing any Actor, Judge, or Editor role causes a large drop, while remaining drift is dominated by harder action-schema errors.The method substantially reduces shortcut-like failures such as goal weakening and object/type drift.
  • Repair dynamics: 46.2% of instances are solved by the initial Actor output, rising to 70.8% after bounded Editor repair.The first two repairs contribute 17.3 of the total 24.6-point gain, while syntax and typing errors are removed first.
  • Solvability–faithfulness gap: The method has a 4.5-point solvability–faithfulness gap, compared with 12.3 points for Qwen-Self-Refine+Solver and 11.3 points for GPT-4o-Self-Refine+Solver.The paired comparison shows that stronger prompting or base models do not fully prevent semantic drift.
  • Judge analysis: A 0.89 overall AUC indicates that the Judge separates faithful outputs from solved-but-unfaithful outputs among solver-successful candidates.Judge scores are low for unsolved outputs, intermediate for solved-but-unfaithful outputs, and high for faithful outputs.

A.5 Training Protocol and Optimization

Training cycles combine Actor generation, Judge scoring, and conditional Editor repair, with shared parameters updated through role-specific objectives and PPO. Solver diagnostics and bounded decoding define the optimization protocol.

  • Training loop: Each training cycle generates an initial specification, queries the solver, trains the Judge from the outcome, and repairs failures with the Editor.Editor refinement is applied only when the initial solver result is unsuccessful.
  • Optimization: PPO uses a 3 × 10^-5 learning rate, batch size 32, and clipping ratio ϵ = 0.2.These are the stated optimization hyperparameters for the reinforcement-learning procedure.
  • Compute protocol: The main run uses approximately 12,000 solver interactions across 8 training cycles and takes approximately 12–16 hours on 8× NVIDIA A100 GPUs.Fast Downward runs on CPU with a fixed 60-second timeout per generated specification.
  • Grounded verification: Fast Downward verifies generated domain–problem pairs within a fixed 60-second timeout and returns structured diagnostics for syntax, typing, operator, and search failures.The binary solver label is one when a valid plan reaches the stated goal within the timeout.
  • Decoding and repair: Training uses stochastic nucleus sampling at temperature 0.7, while evaluation uses greedy decoding and a maximum repair horizon of 5 steps.Successful initial Actor outputs receive no Editor step.
  • Joint optimization: The shared backbone receives gradients from Actor, Judge, and Editor objectives, while each role head receives its corresponding role-specific gradients.The aggregate update combines the three role objectives with λJ = 0.3 and λE = 0.5.

C Judge Calibration Analysis

The analysis formalizes when solver feedback can support correctness and why separate Actor–Editor objectives preserve information about initial specifications.

  • Judge calibration: Under bounded solver–correctness disagreement, the solver label remains a distributional surrogate for task-level correctness.The bounded disagreement condition requires Pr(B≠C)≤ε.
  • Actor–Editor decomposition: A final-only repair objective cannot identify initial specification quality when correct and incorrect initializations are equally repairable.All such initializers can receive the same final return despite differing initial correctness.
  • Actor–Editor decomposition: The Actor receives an initial-stage reward, while the Editor receives a final-stage repair reward for diagnostic-conditioned correction.This decomposition keeps global formalization tied to the initial specification while specializing repair locally.

E Proof of Reward-Hacking Scaling

The reward-hacking analysis compares separate models with a shared-backbone architecture by restricting perturbations that remain neutral to Judge and Editor objectives.

  • Admissible perturbations: Separate models permit Actor perturbations throughout the Actor parameter space, whereas shared backbones restrict neutral perturbations primarily to the Actor head.The shared constraint assumes Judge and Editor constrain shared-backbone directions.
  • Interpretation: Parameter sharing reduces, but does not eliminate, reward hacking by shrinking the role-private degrees of freedom.An effective-rank variant adds k Judge- and Editor-neutral backbone directions, making the relevant dimension h+k.

F.1 Judge–Correctness Consistency

The Judge–correctness analysis connects calibrated solver-success prediction to task correctness under an on-policy bound on specification drift.

  • Calibration: The Judge’s Bayes-optimal score equals the conditional probability of solver success for each task–specification pair.Determinism makes this conditional probability equivalent to the solver outcome for a fixed pair.
  • Consistency bounds: Under bounded drift, the Judge’s expected discrepancy from correctness is controlled by the calibration error plus solver–correctness disagreement.The assumption bounds drift under the policy-induced distribution rather than pointwise.
  • Consistency bounds: A Judge score margin of 2δJ prevents incorrect-over-correct ranking except on drift events with probability at most 2ε.The ranking guarantee follows from calibration error δJ and two candidate-wise drift events.
  • Empirical interpretation: Under solver-produced-plan evaluation, the drift gap equals solvability minus correctness and can be estimated by sampling tasks and generated specifications.The reported empirical gap is typically well below 0.1 in the cited experiments.
  • Actor–Editor consequence: The Actor–Editor formulation aligns initial correctness up to O(ε + δJ), while the Editor weakly improves final solvability.This contrasts with saturated repair, where a monolithic final-only objective admits arbitrarily low initial correctness.

Main result

The main theorem compares reward-hacking scaling under separate models and a shared backbone, showing that shared role constraints make the bound depend on a smaller role-private subspace.

  • Three separate models: Separate-model reward-hacking gain is analyzed over the full Actor parameter space.The separate architecture uses Hsep over R^PA.
  • One shared backbone: Shared-backbone reward-hacking gain is analyzed over perturbations confined to the Actor-head subspace.Judge- and Editor-neutral shared-backbone perturbations vanish under the null-space assumption.
  • Scaling comparison: The shared-architecture bound depends on h rather than backbone dimension p.If k backbone directions remain unconstrained, the relevant scaling becomes h+k.

G.1 Benchmarks

The evaluation uses PlanBench for classical natural-language planning, plus ProntoQA and NATURAL PLAN for zero-shot transfer. Models are assessed through solver execution for PDDL tasks and exact-match benchmark protocols for non-PDDL tasks, without supervised PDDL demonstrations.

  • PlanBench: PlanBench covers BlocksWorld, Mystery BlocksWorld, Logistics, and Gripper, testing symbolic transition reasoning under natural-language descriptions.Mystery BlocksWorld removes lexical familiarity through systematic predicate renaming.
  • Zero-shot transfer benchmarks: ProntoQA and NATURAL PLAN evaluate zero-shot transfer beyond PlanBench domains.ProntoQA tests multi-hop inference, while NATURAL PLAN uses Trip Planning and Calendar Scheduling under constraints.
  • PDDL-based evaluation: PlanBench success requires syntactically valid PDDL, successful planning within 60 seconds, and goal satisfaction.Parsing, grounding, type-checking, or goal-achievement failures count as failures.
  • Non-PDDL evaluation: ProntoQA and NATURAL PLAN use exact-match success under their original benchmark protocols, with consistent answer normalization.The method uses no benchmark-specific supervised examples during training.
  • Framework configuration: The framework uses Qwen2.5-7B in Actor, Judge, and Editor roles for generation, quality scoring, and diagnostic repair.The Actor generates PDDL, the Judge predicts solver-calibrated quality, and the Editor repairs failed specifications.
  • Training and inference: Training uses only symbolic-planner feedback and diagnostics, without human-written PDDL demonstrations or supervised planning traces.At inference, the Editor performs bounded repair after failed solver verification.

H.1 Fair Baseline Comparison under Matched Backbone and Solver Budget

Matched-backbone and matched-budget comparisons isolate the contribution of trained role specialization. Under the same Qwen2.5-7B backbone and K = 6 solver-call cap, the proposed method outperforms prompting and self-refinement baselines while using fewer calls on average and improving faithfulness.

  • Controlled comparison: All controlled methods use Qwen2.5-7B and at most K = 6 solver calls per PlanBench instance.The comparison is designed to control backbone and verifier-access differences.
  • Controlled baselines: Qwen-CoT, Qwen-ToT, Qwen-LLM+P, and Qwen-Self-Refine+Solver provide same-backbone prompting and repair baselines without supervised PDDL training.Self-refinement receives solver diagnostics, while the other prompting variants select among independently generated candidates.
  • Matched-backbone results: 49.8% average success is achieved by Qwen-Self-Refine+Solver, leaving a 21.0-percentage-point gap behind the proposed method.The baseline improves over Qwen-LLM+P by revising syntactic and type-level errors but lacks trained Judge and Editor roles.
  • Budget efficiency: 3.2 solver calls are used on average despite the K = 6 cap, because many instances succeed at initial generation or early repair.Other baselines more frequently exhaust the budget without executable specifications.
  • Interpretation of controlled results: The method’s performance gap is attributed to trained multi-role decomposition rather than backbone choice or a larger verifier budget.The Actor handles global construction, while Judge calibration and Editor repair support the remaining process.
  • Domain behavior: 67% faithful success is retained on Mystery BlocksWorld, where lexical cues are removed.Lower faithful success on Logistics and Gripper reflects sensitivity to typing, movement constraints, and action preconditions or effects.
  • Faithfulness: 66.3% faithful success is reported for the method, while solver-only reinforcement learning increases solvability but produces substantial specification drift.The full model improves solvability and faithful success simultaneously with lower drift.

H.6 Ablation and Drift-Type Discussion

Ablations and drift analyses attribute performance to coordinated Actor, Judge, and Editor roles, while showing that remaining errors are mainly subtle transition-model mistakes. Bounded repair improves cumulative success, and the Judge separates faithful from merely solver-executable outputs.

  • Role ablations: The Actor-only, Actor-plus-Judge, and Actor-plus-Editor ablations fail to match the full model on Mystery BlocksWorld.The full model combines global formalization, calibrated scoring, and bounded local correction.
  • Shared versus separate roles: A separate 3×7B role-model baseline collapses, whereas the shared-backbone design reduces role-private degrees of freedom and stabilizes cross-role credit assignment.The result supports sharing one backbone across the coordinated roles.
  • Drift types: Goal weakening and object/type drift are common shortcut failures in LLM+P and solver-only RL, while the proposed method substantially reduces them.Residual drift is dominated by action-schema drift, indicating subtler transition-model errors.
  • Repair dynamics: 46.2% of instances are solved by the initial Actor, rising to 70.8% after five Editor repairs.The first two repairs contribute 17.3 percentage points, about 70% of the total 24.6-point repair gain.
  • Repair diagnostics: The repair process tracks six diagnostic categories, including syntax, type, schema, precondition/effect, goal/constraint, and unreachable-search errors.A diagnostic heatmap reports the dominant remaining error after each repair step.
  • Solvability–faithfulness gap: The proposed method has the smallest solvability–faithfulness gap, indicating closer alignment between executable PDDL and task-level semantics.The gap measures how tightly solver success aligns with semantic correctness.
  • Judge analysis: Candidate outputs are partitioned into unsolved, solved-but-unfaithful, and solved-and-faithful groups for evaluating Judge behavior.The Judge assigns scores to initial Actor outputs, intermediate repairs, and final specifications.
  • Judge analysis: U < D < F is the Judge’s observed ordering across unsolved, solved-but-unfaithful, and solved-and-faithful candidates.This separation is strongest on BlocksWorld and weaker on Logistics and Gripper.
Loading 2608.21897v1…