Source-linked AI summary
PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models
Ziliang Zhao, Zenan Xu, Shuting Wang, Hongjin Qian, Yan Lei, Minda Hu, Zhao Wang, Shihan Dou, Zhicheng Dou, Pluto Zhou
TL;DR
PlanningBench addresses the limited scalability, diversity, difficulty control, and verification of fixed planning benchmarks. It builds a taxonomy-grounded synthesis framework for self-contained, verifiable instances and finds that models remain challenged while verified-data training transfers to unseen planning tasks.
Problem
Existing planning benchmarks mainly provide fixed instance collections with limited scalable generation, structural difficulty control, automatic verification, and training support.
Method
PlanningBench abstracts real scenarios into more than 30 task types and synthesizes self-contained problems with controllable constraints, difficulty, filtering, and verification checklists.
Results
PlanningBench remains challenging for frontier LLMs, while reinforcement learning on verified data improves performance on unseen planning benchmarks and broader instruction-following tasks.
Takeaways & Limitations
PlanningBench provides a controllable source of planning data for evaluating and training generalizable planning abilities.
Takeaways & Limitations
PlanningBench targets text-based problems where all information needed to construct and verify a plan is provided in the input.
Abstract
from arXiv · showhide
Planning is a fundamental capability for large language models (LLMs) because such complex tasks require models to coordinate goals, constraints, resources, and long-term consequences into executable and verifiable solutions. Existing planning benchmarks, however, usually treat planning data as fixed collections of instances rather than controllable generation targets. This limits scenario coverage, ties difficulty to surface-level proxies rather than structural sources, and offers limited support for scalable generation, automatic verification, or planning-oriented training. We introduce PlanningBench, a framework for generating scalable, diverse, and verifiable planning data for both evaluation and training. PlanningBench starts from real planning scenarios and abstracts practical workflows into a structured taxonomy of more than 30 task types, subtasks, constraint families, and difficulty factors. Guided by this taxonomy, a constraint-driven synthesis pipeline instantiates self-contained planning problems with adaptive difficulty control, quality filtering, and instance-level verification checklists. This shifts planning data construction from fixed benchmark collection to controllable generation while preserving realistic task grounding. We use PlanningBench to evaluate open-source and closed-source frontier LLMs, and find that current models still struggle to produce complete solutions under coupled constraints. Beyond evaluation, reinforcement learning on verified PlanningBench data improves performance on unseen planning benchmarks and broader instruction-following tasks. Further analysis suggests that determinate or well-specified optimal solutions provide clearer reward signals and more stable training dynamics. Overall, PlanningBench provides a controllable source of planning data for diagnosing and improving generalizable planning abilities in LLMs.
1 Introduction
PlanningBench addresses limitations of fixed planning benchmarks by generating scalable, diverse, and verifiable data from real planning scenarios. It supports both evaluation and training while exposing persistent weaknesses in current LLM planning.
- Motivation: Existing benchmarks often use fixed task collections, limiting scalable generation, diversity, controlled difficulty, and automatic verification.Their difficulty is also often tied to surface-level proxies rather than structural planning factors.
- Framework: PlanningBench abstracts real scenarios into a taxonomy covering more than 30 planning tasks, subtasks, constraint families, and difficulty factors.This preserves realistic task grounding while enabling systematic composition of diverse instances.
- Framework: Its constraint-driven pipeline generates self-contained problems, applies adaptive difficulty enhancement and quality filtering, and pairs instances with verification checklists.These mechanisms support controllable diversity across domains, subtasks, constraints, and solution structures.
- Evaluation: Current frontier LLMs still struggle with complete solutions under coupled constraints, and PlanningBench remains challenging even for the strongest evaluated models.The suite also reveals how scenario structure and constraint complexity affect planning success.
- Training: Verified PlanningBench data supports planning-oriented reinforcement learning, improving performance on unseen planning benchmarks and broader instruction-following tasks.The authors further identify determinate or well-specified optimal solutions as important for stable and transferable training.
2 Related Work
PlanningBench complements prior planning benchmarks by treating data construction as controllable generation rather than fixed benchmark collection. Its taxonomy and verification checklists target broader coverage, structural difficulty control, and reuse for training.
- Existing benchmarks: Prior planning benchmarks span realistic travel, retrieval, dynamic, multimodal, web-based, and other planning settings.They provide varied evaluation environments and task formulations.
- PlanningBench: PlanningBench abstracts reusable task and constraint structures from real scenarios and synthesizes self-contained instances through task-specific composition and hierarchical difficulty control.Its taxonomy covers more than 30 task types.
- PlanningBench: Verification checklists let the same generated data support complete-solution evaluation and planning-oriented training.This distinguishes the framework from another fixed evaluation set.
- Research focus: Related methods mainly strengthen planners, agents, or tool-use systems through decomposition, collaboration, repair, reinforcement learning, and synthesized experience.PlanningBench instead focuses on the controllable construction of planning data.
3 Data Construction
PlanningBench constructs self-contained, verifiable planning instances by combining real-scenario taxonomies with constraint-driven synthesis and adaptive difficulty control. Its closed-loop pipeline generates, evaluates, filters, and revises instances for scalable data creation.
- Definition and scope: PlanningBench targets text-based planning problems where all information needed to construct and verify a plan is provided in the input.Each instance consists of a self-contained planning task and a verification checklist.
- Taxonomy construction: Construction first abstracts representative real scenarios into reusable task and constraint taxonomies covering planning structures, subtasks, and difficulty factors.The taxonomy is organized by planning structure rather than surface application labels.
- Quality control: Candidate instances are filtered and revised for clarity, consistency, and verifiability before entering the data pool for evaluation and planning-oriented training.The process preserves realistic grounding while converting fixed collection into controllable generation.
- Taxonomy construction: The task taxonomy contains six structural families, including scheduling, allocation, workforce scheduling, routing, project operations, and emergency response.Each family contains concrete tasks divided into 5–10 subtasks on average.
- Constraint design: Constraints are modeled jointly with task types and organized as general, task-specific, and specialized stateful requirements.This coupling makes feasibility and optimization requirements reusable during synthesis rather than post-hoc annotations.
- Difficulty control: Difficulty levels progress from basic feasibility requirements to medium optimization objectives and hard structural demands such as infeasibility recognition and robustness.The pipeline samples task variants and constraints, then applies closed-loop enhancement.
- Closed-loop synthesis: A Generator creates candidate problems, a Responder attempts solutions, and a Critic verifies them against checklists, producing verification scores and all-pass indicators.Solved instances trigger higher-difficulty constraint compositions, while unsolved instances remain challenging under the current configuration.
4 Experiments
PlanningBench evaluates whether language models can produce complete, globally consistent plans under coupled constraints, while diagnosing partial compliance and failure sources. Results show substantial difficulty across models, task structures, information loads, and semantic error types.
- Evaluation Protocol: PlanningBench evaluates complete-solution success and partial checklist satisfaction to distinguish globally valid plans from locally compliant outputs.All-pass requires satisfying every checklist item in one response, whereas Avg-pass measures the fraction satisfied.
- Model Performance: 63.17% All-pass and 92.35% Avg-pass make GPT-5.4-xhigh the strongest evaluated model, yet the suite remains far from saturated.Gemini-3-1-pro follows with 53.25% All-pass and 88.36% Avg-pass; even the best model fails the full checklist on more than one third of instances.
- Model Performance: 90.03% Avg-pass but 58.09% All-pass for GPT-5.4-medium illustrates that satisfying many local requirements does not ensure a complete, globally consistent plan.Seed-2.0-pro-high shows a similar gap, with 84.02% Avg-pass and 44.33% All-pass.
- Difficulty Factors: 80.0% and 69.2% All-pass are GPT-5.4-xhigh’s best scores on Hiring Plan and Material Allocation, while Gemini-3-1-pro peaks at 72.7% on Student Grouping.Cross-model ranges reach 80.0%–6.7% on Hiring Plan and 53.9%–7.1% on Power Dispatch Planning.
- Difficulty Factors: 40.0% All-pass in the 15+ checklist-item group versus 63.4% for 2–6 items shows that more checklist requirements make complete success harder.GPT-5.4-xhigh declines from 64.0% in Very Short prompts to 50.6% in Very Long prompts; Gemini-3-1-pro drops from 61.5% to 43.8%.
- Error Analysis: 60.9%–83.5% of semantic errors are Wrong Calculation / Assignment, exceeding Constraint Omitted, State Tracking, Format / Structure, and Missing Rationale errors.The analysis excludes exact refusals and blank or no-answer outputs, assigning each remaining failed case one primary semantic error type.
4.2 PlanningBench as Training Data
PlanningBench data supports both transfer evaluation and reinforcement-learning analysis. Training improves external planning and instruction-following performance, while determinate optima produce more stable optimization dynamics.
- 4.2.1 Training Setup: GRPO on 300 PlanningBench instances evaluates transfer to external planning and general instruction-following benchmarks.The setup uses Qwen-A3B-30B and compares Base Model, Syn-PlanningBench, Syn-NotDetOptimal, and Human-Authored data.
- 4.2.2 Generalization to External Planning Benchmarks: Syn-PlanningBench improves ChinaTravel Avg-pass from 84.88% to 88.12% and All-pass from 50.92% to 58.36%.On ChinaTravel-Human, All-pass rises from 86.76% to 93.18%.
- 4.2.2 Generalization to External Planning Benchmarks: Syn-PlanningBench transfers more strongly than Human-Authored data, reaching 46.86% average All-pass on TravelPlanner versus 33.86%.On ChinaTravel, the corresponding average All-pass improvements are 7.44 points and 1.49 points.
- 4.2.3 Transfer to General Instruction-following Benchmarks: Syn-PlanningBench obtains the best or tied-best scores on all three general instruction-following benchmarks, raising the average from 38.74% to 45.80%.Collie increases from 38.33% to 53.17%.
- 4.2.3 Transfer to General Instruction-following Benchmarks: Syn-NotDetOptimal yields only a 0.75-point average gain and slightly decreases Inverse IFEval performance.The comparison suggests that broader transfer depends on clearer optimality and verification signals, without implying uniform gains across general capabilities.
- 4.2.4 Training Dynamics and the Role of Determinate Optima: Syn-PlanningBench has the fastest-decreasing and lowest-converging solve-none ratio, alongside a smoother critic-reward curve.Syn-NotDetOptimal maintains a low solve-all ratio, while Human-Authored performs between the two synthetic settings.
- 4.2.4 Training Dynamics and the Role of Determinate Optima: Determinate or well-specified optima provide clearer reward direction for complete planning success and more stable, transferable reinforcement learning.Permissive rewards may improve partial checklist satisfaction without reliably producing complete solutions.
5 Conclusion
PlanningBench combines controllable, verifiable data construction with challenging evaluation and training transfer. Its results support determinate rewards as a useful condition for more stable planning-oriented learning.
- 5 Conclusion: PlanningBench generates scalable, diverse, and verifiable data through reusable task structures, adaptive difficulty control, automatic verification, and quality filtering.The framework starts from real planning scenarios and supports both evaluation and planning-oriented training.
- 5 Conclusion: Training on verified PlanningBench data improves transfer to external planning and broader instruction-following benchmarks.The conclusion also reports substantial remaining gaps for the strongest models under All-pass evaluation.
- 5 Conclusion: Determinate or well-specified optimal solutions provide more stable and directional reward signals.The reported failures primarily concern constrained reasoning and global consistency rather than formatting.
A Task Taxonomy Summary
The task taxonomy summarizes PlanningBench’s planning-task coverage by organizing task families, representative subtasks, and the abilities each task tests.
- A Task Taxonomy Summary: Table 6 lists PlanningBench task families, representative subtasks, and the primary abilities targeted by each task.It supplements the main discussion of task construction.
B Constraint Taxonomy Summary
PlanningBench operationalizes difficulty and verifiability through reusable general constraints and task-specific constraints. The listed examples span feasibility, dependencies, resources, timing, recovery, coordination, and multi-objective requirements.
- B Constraint Taxonomy Summary: Tables 7 and 8 distinguish reusable general constraints from task-specific constraints for concrete planning scenarios.Together, they show how PlanningBench operationalizes difficulty and verifiability.
- IT System Maintenance Planning: IT maintenance examples include maintenance windows, rollback conditions, canary rollout order, monitoring windows, cross-team coordination, and version-failure rollback.The constraints encode temporal, dependency, coordination, and recovery requirements.
- Hard Constraints: Hard constraints require explicit recovery, disruption minimization, infeasibility diagnosis, multi-objective comparison, and coupled-failure handling.Examples include failure-to-recovery switching, minimum business disruption, bottleneck identification, and cross-region consistency recovery.
- Basic Constraints: Basic constraints cover availability, capacity, precedence, time boundaries, role matching, and quantitative limits.Examples include meeting attendance, room capacity, valid duration, no time conflict, and service-impact boundaries.
- Medium Constraints: Medium constraints refine solution quality through fragmentation reduction, fairness, priority, phased strategies, validation, coordination, and rollback.These constraints require explicit coordination or recovery beyond basic feasibility.
C Implementation Details for Constraint Sampling
The synthesis pipeline samples constraint compositions by task and subtask, checks compatibility, generates problem-checklist pairs, and projects adaptive difficulty updates back into admissible counts.
- Initial constraint-count sampling: The Generator samples basic, medium, and hard constraints for each task and subtask variant from task-specific pools.
- Initial constraint-count sampling: The initial priors ensure at least one basic constraint, usually include medium requirements, and introduce hard constraints sparsely.Closed-loop difficulty enhancement can later shift samples toward harder compositions.
- Subset construction: After drawing counts, the Generator samples constraint subsets without replacement and checks them for duplicate, contradictory, or underspecified requirements.Stateful constraints are sampled separately when required by the task template.
- Problem generation: The sampled constraints and a variable controlling wording, numerical instantiation, and context define a generation specification that produces a problem-checklist pair.The pair is generated as (x, c) = G(r).
- Projection after difficulty resampling: After adaptive resampling, the updated difficulty distribution is projected onto admissible count ranges to preserve basic, medium, and hard limits while controlling problem size.The projection reflects the updated difficulty distribution.
D Human Quality-control Categories
Human quality control assigns synthesized samples to four categories based on usability and recoverability, with most audited samples usable immediately or after minor revision.
- Quality-control categories: Annotators classify samples as usable without modification, requiring minor revision with usable source data, requiring source-data correction, or discard.The categories distinguish recoverable quality issues from irrecoverable ambiguity, inconsistency, or verification failure.
- Audit outcomes: 86.15% of audited samples fall into categories (1) and (2), while 13.85% require category-(3) revision.
- Audit outcomes: No audited sample is directly assigned to the discard category.
- Audit outcomes: Most revisions address missing checklist items, incomplete constraint checks, ambiguous prompts, unclear references, missing rules, boundary conditions, or prompt-answer inconsistencies.
E Break Case
The break-case section presents a production-planning example with a gold solution, a model response, and error analysis to inspect global-consistency failures penalized by PlanningBench.
- Break Case: Table 9 uses a production-planning example to compare a gold solution with a model response and corresponding error analysis.
- Break Case: The example illustrates global-consistency mistakes that PlanningBench penalizes.
F Training Details
The training setup uses GRPO-based reinforcement learning with specified optimization, rollout, and inference parameters.
- Training configuration: GRPO training uses batch size 128 for 50 epochs, actor learning rate 2 × 10−6, KL coefficient 0.001, and 8-way rollout sampling.
- Inference configuration: Inference uses temperature=0.7, top-p=0.6, and top-k=20.
G Prompt Templates for Task Synthesis and Critic Evaluation
The appendix specifies prompt templates for creating realistic, executable planning tasks with concrete constraints and binary verification checklists, then instructs a critic to score answers systematically against those requirements.
- G Prompt Templates for Task Synthesis and Critic Evaluation: The synthesis template produces planning tasks and corresponding checklists that can be edited manually before critic-based evaluation.
- G.1 Prompt for Synthesizing the Task and Checklist: Tasks must require genuine planning, scheduling, allocation, dispatching, or arrangement rather than general advice.
- G.1 Prompt for Synthesizing the Task and Checklist: Tasks should be realistic, concrete, executable, and verifiable, while remaining grounded in scenario information and requiring an executable plan from explicit input data.
- G.1 Prompt for Synthesizing the Task and Checklist: Task inputs should include structured elements such as objects, resources, time ranges, capacities, budgets, conflicts, dependencies, priorities, and exceptions.
- G.1 Prompt for Synthesizing the Task and Checklist: Checklist criteria use binary scoring, assigning 1 only when all conditions are satisfied and 0 when any key condition is violated.
- G.1 Prompt for Synthesizing the Task and Checklist: Checklist requirements should directly verify task outputs, coverage, constraints, formats, objectives, and any requested recovery or infeasibility handling.
- G.2 Prompt for GPT-OSS-120B Used as the Critic: The critic evaluates every checklist requirement jointly with equal weight and outputs a scoring rationale, satisfaction-status list, and final score.
- G.2 Prompt for GPT-OSS-120B Used as the Critic: The grading procedure requires requirement analysis, one-by-one answer checks, and self-reflection for completeness, strictness, consistency, and objectivity.