Source-linked AI summary
PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, Subbarao Kambhampati
TL;DR
LLM planning claims often rely on commonsense tasks that do not clearly distinguish planning from knowledge retrieval, motivating systematic and diverse benchmarks. PlanBench provides an extensible, automated suite based on automated-planning domains and evaluates multiple planning capabilities; its initial studies find subpar performance even in simple common-sense planning domains. The benchmark is intended as a resource for evaluating current and future LLMs.
Problem
Existing LLM planning evaluations often use commonsense tasks, making it difficult to determine whether models genuinely plan or retrieve world knowledge.
Method
PlanBench combines automated-planning domains with a curriculum of planning tasks, generated prompts, domain models, planners, translators, and plan validators.
Results
LLMs show subpar performance across many critical planning capabilities, including plan generation, even in simple common-sense planning domains.
Takeaways & Limitations
PlanBench provides an extensible assessment suite for comparing current and future LLMs on planning and reasoning about actions and change.
Takeaways & Limitations
The reported experiments primarily use one-shot natural-language prompts and focus on GPT-4, InstructGPT3, and the Blocksworld domain.
Abstract
from arXiv · showhide
Generating plans of action, and reasoning about change have long been considered a core competence of intelligent agents. It is thus no surprise that evaluating the planning and reasoning capabilities of large language models (LLMs) has become a hot topic of research. Most claims about LLM planning capabilities are however based on common sense tasks-where it becomes hard to tell whether LLMs are planning or merely retrieving from their vast world knowledge. There is a strong need for systematic and extensible planning benchmarks with sufficient diversity to evaluate whether LLMs have innate planning capabilities. Motivated by this, we propose PlanBench, an extensible benchmark suite based on the kinds of domains used in the automated planning community, especially in the International Planning Competition, to test the capabilities of LLMs in planning or reasoning about actions and change. PlanBench provides sufficient diversity in both the task domains and the specific planning capabilities. Our studies also show that on many critical capabilities-including plan generation-LLM performance falls quite short, even with the SOTA models. PlanBench can thus function as a useful marker of progress of LLMs in planning and reasoning.
1 Introduction
PlanBench addresses the need for systematic evaluation of whether LLMs can plan and reason about actions and change, rather than merely retrieve commonsense knowledge. It introduces an extensible benchmark and automated assessment framework grounded in automated-planning domains.
- Planning is framed as a core reasoning capability involving actions that transform an initial state into a desired world state.
- PlanBench uses domains from the automated-planning community, especially International Planning Competitions, to test LLM reasoning about actions and change.
- The benchmark provides a curriculum of distinct planning tasks and code for automatically generating queries across those tasks.
- PlanBench initializes its dataset with Blocksworld and Logistics, including obfuscated variants using misleading words or random alphanumeric strings.The initial dataset contains approximately 26,250 prompts.
- Unlike subjective demonstrations of isolated failures, the framework automatically generates queries and validates LLM answers using planning models and tools.
2 Related Work
Related work has examined LLM reasoning and planning roles, but PlanBench emphasizes systematic, extensible, and mechanistic assessment across multiple planning capabilities.
- Existing reasoning benchmarks include BIG-BENCH, GSM8K, AQUA, and SVAMP, while PlanBench targets planning abilities specifically.
- LLMs have been studied as plan generators, heuristic providers, knowledge extractors, and systems that incorporate environmental or user feedback.
- PlanBench presents prompts as few-shot examples and uses domain descriptions to constrain possible actions.
3 Background
The paper grounds PlanBench in classical deterministic planning, where domains, initial states, goals, fluents, and action preconditions and effects define state transitions.
- Classical planning is a goal-directed deterministic formalism for solving basic planning problems.
- A planning problem is represented as P = ⟨D, I, G⟩, consisting of a domain, initial state, and goal specification.
- The domain specifies fluents and actions, while each action has preconditions determining executability and effects determining the resulting state.
- For the put-down action, holding a block is required, and execution makes the block clear and on the table while emptying the arm.
- The formalism can be extended with object typing, richer preconditions, conditional effects, and other planning representations.
4 Assessment Architecture
PlanBench separates domain-independent testing infrastructure from domain-dependent models and generators, connecting symbolic planning representations to natural-language prompts and validated LLM outputs.
- The framework has domain-independent components supplied by PlanBench and domain-dependent components developed for each tested domain.
- The domain-independent layer uses a planner and plan verifier to create curriculum test instances and validate LLM-generated solutions.
- A test-case-specific component turns generated planning problems into prompt content, while planners and validators supply or check relevant information.
- The domain-dependent layer contains a lifted domain model, a problem generator, and a translator.
- The problem generator creates objects, initial states, and goals, and can encode distributional requirements for evaluation.
- The translator converts symbolic models to natural language and parses generated plans back into structured forms for plan validation.
5 Current Curriculum for Testing
PlanBench uses a few-shot curriculum to test distinct planning and action-change reasoning capabilities across IPC-style domains. It includes planning tasks, auxiliary reasoning tasks, and robustness tests such as plan generalization and goal reformulation.
- The curriculum uses few-shot examples, domain descriptions, and a new instance requiring the LLM to produce or assess a completion.
- PlanBench evaluates eight capabilities, including plan generation, cost-optimal planning, plan verification, execution reasoning, and robustness to goal reformulation.
- Plan Generation: Plan generation presents example problems with plans before asking for a plan for a new initial state and goal.
- Cost-Optimal Planning: Cost-optimal planning adds action costs and asks for plans minimizing time, money, or another stated cost.
- Auxiliary Tasks: Plan verification and execution reasoning separately assess whether plans achieve goals and whether LLMs can predict resulting states after action sequences.
- Plan Generalization: Plan generalization tests whether LLMs infer patterns from program-generated example plans and reuse them for new problems.
6 Dataset Details
The initial PlanBench dataset covers Blocksworld and Logistics, with varied problem sizes and obfuscated domain terminology. It contains approximately 26,250 prompts across domains and test cases.
- PlanBench is initialized with two IPC domains, Blocksworld and Logistics, while supporting curriculum execution on new domains.
- Blocksworld: Blocksworld contains 600 instances varying in object count and optimal plan length, with one hand and stack-order goals.
- Logistics: Logistics contains 285 instances involving packages, trucks, planes, cities, and locations, with distributions of objects and optimal plan lengths.
- Obfuscation: Obfuscation replaces familiar domain names, predicates, and actions with misleading words or random alphanumeric strings.
- The dataset contains approximately 26,250 prompts across test cases, domains, and obfuscated versions.
7 Specimen Evaluation of PlanBench
Initial evaluation on Blocksworld compares GPT-4 with InstructGPT3 across PlanBench tasks. GPT-4 performs best on auxiliary goal-reformulation tasks but remains weak on most capabilities.
- GPT-4 and InstructGPT3 were evaluated on PlanBench using the Blocksworld domain as initial baseline models.
- GPT-4 achieved its strongest results on auxiliary goal-reformulation tasks, yet performed below a satisfactory level on most Blocksworld test cases.
- The reported overall performance indicates that these LLMs were ineffective at reasoning about actions and change at the time of evaluation.
8 Conclusion and Future Work
The paper presents PlanBench as an extensible suite for evaluating distinct aspects of LLM planning and action-change reasoning. It reports subpar performance in simple common-sense planning domains and identifies several future extensions.
- PlanBench consists of test cases that each evaluate a central aspect of planning or reasoning about actions and change.
- LLMs showed subpar performance even in simple common-sense planning domains.
- Future Work: Future work includes partial-correctness metrics, additional IPC domains, and evaluation across more models, including fine-tuned systems.
A.1 Broader Impact
PlanBench covers a broad range of planning and reasoning tasks while avoiding personally identifiable and privacy-related information. However, benchmark performance alone does not guarantee correct or safe plans in external deployment.
- PlanBench covers a broad range of tasks related to planning and reasoning.
- The benchmark contains no personally identifiable or privacy-related information.
- Performance on PlanBench alone does not guarantee that externally deployed models will produce correct or safe plans.
A.2 Additional details on experiments
Experiments used deterministic OpenAI API settings, evaluated GPT-4 with an 8k context window, and relied on Fast-Downward and VAL for planning and validation.
- All experiments used the OpenAI API with default parameters except temperature, which was set to 0 for deterministic outputs.
- GPT-4 was evaluated with an 8k context window using a version accessed between March and June.
- Fast-Downward served as the planner, while VAL served as the plan validator.
A.3 Obfuscation experiments
Obfuscating planning domains substantially reduced GPT-4 and Instruct-GPT3 performance, indicating that Blocksworld success was more consistent with pattern matching than robust reasoning.
- GPT-4 and Instruct-GPT3 performance decreases significantly when domains are deceptively or randomly obfuscated.
- Table 2 reports GPT-4 and Instruct-GPT3 results for the Plan Generation test case.
- The Blocksworld performance was more likely due to pattern matching than reasoning robust to domain obfuscation.
A.4.1 Comparison to the dataset
Figure 4 compares Blocksworld instances and GPT-4’s correct plans across optimal plan lengths. Smaller optimal plan length does not necessarily correspond to higher LLM performance.
- The left graph distributes Blocksworld instances by optimal plan length, while the right distributes GPT-4 correct plans by optimal plan length.
- Comparing the distributions shows that shallower instances do not necessarily yield higher LLM performance.
- The result differs from classical-planner intuition because LLMs predict the next token using model weights and context for each instance.
A.4.2 Failure modes analysis
The analysis examines GPT-4’s plan-generation failures using more forgiving validity evaluations based on delete and precondition relaxations.
- GPT-4 failure cases were examined through a more forgiving evaluation of generated-plan validity.The evaluation used domain-model relaxations from automated planning.
- Delete relaxation ignores action delete conditions when evaluating plans.
- Precondition relaxation ignores all action preconditions, treating every action as executable in any state.
A.5 Plan Generation Prompts
Plan generation prompts test whether models can produce valid action sequences across Blocksworld, Logistics, and Mystery Blocksworld domains. The examples specify initial states, goals, actions, restrictions, and candidate plans for comparison with ground truth.
- A.5.1 Blocksworld: Blocksworld prompts provide block arrangements, goals, permitted actions, and action restrictions before requesting a plan.Restrictions include hand occupancy, block clearance, and placement conditions.
- A.5.1 Blocksworld: A second Blocksworld example compares a GPT-4 response containing extra actions with a shorter ground-truth plan.The response and ground truth both target placing the orange block on the red block.
- A.5.1 Blocksworld: GPT-4’s first Blocksworld example matches the ground-truth plan after extracting actions from the response.The extracted plan is marked SUCCESS.
- A.5.2 Logistics: Logistics prompts define package transport using trucks within cities and airplanes between cities.The domain includes locations, cities, vehicles, packages, action restrictions, and explicit goals.
- A.5.3 Mystery Blocksworld: Mystery Blocksworld replaces familiar action semantics with named actions whose preconditions and effects are explicitly described.The example goal is for object c to crave object b, with a proposed plan and ground-truth sequence.
- A.5.3 Mystery Blocksworld: The Mystery Blocksworld example’s ground-truth plan uses Feast, Succumb, Attack, and Overcome actions in sequence.
A.6 Cost Optimal Planning Prompts
Cost-optimal planning prompts extend the domains with action durations and ask for plans that minimize execution time. The examples compare GPT-4 plans and timings with ground-truth plans across Blocksworld, Logistics, and Mystery Blocksworld.
- A.6.1 Blocksworld: Cost-optimal Blocksworld prompts add action durations and ask models to minimize the time needed to reach each goal.Each listed Blocksworld action takes 1 minute.
- A.6.1 Blocksworld: GPT-4 produces a 4-minute Blocksworld plan for placing the orange block on the blue block.
- A.6.1 Blocksworld: GPT-4 produces a 6-minute plan for placing the orange block on the red block, while the extracted plan includes additional actions beyond the ground truth.
- A.6.2 Logistics: Cost-optimal Logistics prompts assign durations to loading, unloading, driving, and flying actions while retaining domain restrictions.Loading and unloading take 1 minute, driving takes 2 minutes, and flying takes 5 minutes.
- A.6.2 Logistics: The cost-optimal Logistics example asks GPT-4 to transport package_0 between two airport locations from an initial state.
- A.6.2 Logistics: The GPT-4 Logistics plan first flies an airplane to the package, then loads, flies back, and unloads it in 4 minutes.
- A.6.3 Mystery Blocksworld: Cost-optimal Mystery Blocksworld assigns 1 minute to each of Attack, Feast, Succumb, and Overcome actions.The domain retains explicit preconditions and effects for these actions.
A.7 Plan Verification Promptss
Plan verification examples test whether generated plans satisfy action preconditions and goal conditions across Blocksworld, Logistics, and Mystery Blocksworld. The examples include valid plans, invalid plans with diagnosed failures, and comparisons between GPT-4 outputs and ground-truth plans.
- A.7.1 Blocksworld: A.7.1 Blocksworld verification accepts a plan that rearranges the blocks to place orange on red.The verified plan first unstacked and put down blue, then picked up orange and stacked it on red.
- A.7.1 Blocksworld: A.7.1 Blocksworld verification rejects plans when a goal remains unmet or an action precondition is false.Failures include leaving orange off white and attempting to pick up yellow without the required table, clear, and hand conditions.
- A.7.1 Blocksworld: A.7.1 Blocksworld also verifies that GPT-4 correctly identifies an invalid plan whose orange-on-red goal is unmet.The response matches the ground-truth diagnosis rather than proposing a valid execution sequence.
- A.7.2 Logistics: A.7.2 Logistics verification identifies invalid plans through unmet action preconditions or unmet delivery goals.Examples flag unloading package_3 from a truck that does not contain it and failing to place package_1 at location_2_0.
- A.8.3 Mystery Blocksworld: A.8.3 Mystery Blocksworld represents a ground-truth plan with domain-specific predicates such as province, pain, planet, and craves.The passage shows the symbolic plan rather than a verification judgment.
- A.9.1 Blocksworld: A.9.1 Blocksworld replanning produces a shorter sequence after the state changes, or no further actions when the goal is already achieved.The replanned examples either move orange from red to blue or report that orange is already on red.
- A.10.1 Blocksworld: A.10.1 Blocksworld exposes extra actions in an LLM plan by contrasting the generated sequence with the shorter ground-truth plan.The generated plan includes blue, red, black, cyan, silver, and violet operations, while the ground truth retains only cyan and silver operations; another example separates the matching and extraneous actions.
- A.11 Plan Reuse: A.11.1, A.11.2, and A.11.3 examples show plan reuse across Blocksworld, Logistics, and Mystery Blocksworld, with generated plans containing actions beyond the ground truth.The ground truths stop after yellow is put down, after the airplane flies, or after feast and succumb, respectively.