PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change

• PlanBench evaluates LLMs’ ability to reason about actions and change in common-sense planning tasks, motivated by planning’s role in intelligence and potential applications such as automatic code generation.The benchmark draws on domains from the automated planning community and International Planning Competitions (IPC).Source: §1 · p. 2
• The initial benchmark covers Blocksworld and Logistics, their obfuscated variants, and approximately 26,250 prompts, with additional IPC domains and tasks planned.Obfuscations use either misleading words or random alphanumeric strings.Source: §1 · p. 2
• Rather than highlighting isolated failures, the framework provides a reusable assessment curriculum spanning related planning tasks and supports evaluation of models including Instruct-GPT3 [22] and GPT-4.The framework is intended for analyzing other LLMs, including models fine-tuned for planning tasks.Source: §1 · p. 2
• PlanBench reduces subjectivity by mechanistically generating queries and validating answers with automated planning models and tools.This distinguishes the framework from earlier analyses that examined a few example reasoning cases.Source: §1 · p. 2

Related work

• PlanBench is introduced as the first benchmark specifically designed to evaluate emerging LLM planning abilities, expanding substantially with additional test cases and domains.The authors distinguish it from existing benchmarks such as BIG-BENCH [29], GSM8K [3], AQUA [18], SVAMP [23], CommonsenseQA [31], and StrategyQA [5], which involve simpler, shallower reasoning.Source: §2 · p. 2
• Prior work has used LLMs to generate plans [11] [33], produce heuristics [33] [1], extract planning knowledge [7], and improve plans through user or environmental feedback [12] [35] [25].In Say-Can [1], the LLM scores executable robot actions, functioning as a planning heuristic.Source: §2 · pp. 2–3

PlanBench’s assessment framework

• PlanBench evaluates a wide variety of planning capabilities through multiple tasks, each targeting a distinct aspect of reasoning about actions and change.Tasks use few-shot prompts with an instance and example completion before requesting completion on a new instance.Source: §2 · p. 3
• Explicit domain descriptions constrain possible actions, introduce unforeseen limitations and constraints, and force models beyond repeating domain information from training data.This prompt conditioning is intended to test whether general systems can be customized for a specific domain of interest.Source: §2 · p. 3

• Classical planning problems are deterministic, goal-directed tasks represented as P = ⟨D, I, G⟩, where D is the domain, I the initial state, and G the goal specification.Source: §3 · p. 3
• A planning domain defines fluents and actions whose preconditions determine executability, while add and delete effects specify how actions change the state.Source: §3 · p. 3
• Actions may be represented in lifted form with variables or grounded by replacing variables with objects; associated costs also permit defining optimal plans.Source: §3 · p. 3
• The Blocksworld put-down action illustrates that holding a block is required before execution, after which the block is clear and on the table and the arm is empty.Source: §3 · p. 3
• The basic formalism can be extended with object typing, richer preconditions, conditional effects, and broader planning formalisms.Source: §3 · p. 3

• The framework combines domain-independent and domain-dependent components, with the former supplied generally and the latter developed for each tested domain.Source: §4 · p. 4
• The domain-independent architecture uses a planner, problem generator interface, translator, and plan validator to create test cases and verify LLM solutions.It operates primarily on symbolic models such as PDDL, while test-case-specific components craft prompts from generated problems.Source: §4 · p. 4
• Each domain-dependent component contains a lifted domain model, a problem generator, and a translator.The lifted model defines available actions, predicates, and object types independently of the specific objects in any problem instance.Source: §4 · p. 4
• The problem generator creates random instances with objects, initial states, and goals, while allowing desired test distributions to be encoded.Source: §4 · p. 4
• A template-based translator converts symbolic states, actions, prompts, and plans into natural language and parses line-separated LLM plans back into validator-compatible action forms.Parsing uses template matching or maps verbs and nouns to actions and objects, enabling validation by tools such as.Source: §4 · pp. 4–5

Curriculum overview

• The curriculum covers plan generation, cost-optimal planning, verification, execution reasoning, goal-reformulation robustness, plan reuse, replanning, and plan generalization.Each test uses few-shot examples and evaluates a distinct capability involving actions and change.Source: §5 · p. 5

Planning tasks

• Plan generation evaluates whether an LLM produces a valid goal-achieving plan, while cost-optimal planning additionally checks whether its plan has optimal cost.Prompts provide lifted domains, example problems and plans, and a final problem; generated plans are extracted using a dedicated plan-end tag.Source: §5 · pp. 5–7

Verification and execution reasoning

• Plan verification asks whether a candidate plan executes and reaches its goal, requiring the first failure and a missing precondition or goal condition when applicable.Execution reasoning instead predicts the resulting state of always-executable action sequences, which is checked with a plan executor.Source: §5 · p. 7

Reuse and reformulation

• Goal-reformulation robustness tests recognition of equivalent or overlapping goals, while plan reuse tests whether a previously seen plan or prefix solves a new problem.Both retain the same initial state across examples and evaluate the resulting plan as in plan generation.Source: §5 · p. 7

Replanning

• Replanning evaluates whether an LLM produces a valid goal-achieving plan after an unexpected change makes the original execution state different.Training instances show an original plan, its execution, an unexpected fact change, and a replacement plan; event types are constrained by domain.Source: §5 · p. 7

Plan generalization

• Plan generalization tests whether an LLM extracts procedural patterns from traces generated by fixed programs containing loops or conditionals and applies them to new problems.Problems must be curated so generalized behavior solves them by adding iterations of the pattern.Source: §5 · pp. 7–8

Blocksworld

• The dataset includes 600 Blocksworld instances varying in object count and optimal plan length, plus 500 separate generalization instances.Replanning changes a state after executing a random plan prefix by stacking a held block onto another clear block, then asks the LLM to replan.Source: §6 · p. 8, Figure 2 · p. 8

Logistics

• The Logistics set contains 285 instances varying in domain objects and optimal plan length, with additional generalization and replanning configurations.Generalization includes chained package movements and intercity flights; replanning relocates a package from a truck or airplane to a non-goal location before replanning.Source: §6 · p. 8, Figure 3 · p. 9

Obfuscation of domains

• Obfuscated domain versions replace action, predicate, and object names with misleading words or random strings to test reliance on domain models rather than commonsense associations.The benchmark also provides code for arbitrary obfuscations of these and future domains; standard planners treat original and obfuscated domains identically.Source: §6 · p. 9

Dataset scope

• Across domains, test cases, and obfuscated versions, the dataset comprises approximately 26,250 prompts.The paper next reports GPT-4 and InstructGPT-3 results on Blocksworld.Source: §6 · p. 9

Evaluation setup

• GPT-4 and InstructGPT3 were evaluated on PlanBench’s Blocksworld domain using the OpenAI API.The evaluation provided initial baselines for the benchmark.Source: §7 · p. 9

Results

• GPT-4 achieved the strongest overall performance, yet both models fell short on most Blocksworld test cases.The best results within each model occurred on auxiliary goal reformulation tests.Source: §7 · p. 9

Interpretation and companion study

• The benchmark results indicate that current LLMs are largely ineffective at reasoning about actions and change, making PlanBench a useful progress marker.A companion study [33] further examined plan generation, prompting, failure modes, and heuristic-role performance shifts.Source: §7 · p. 9

Contribution

• PlanBench evaluates central aspects of planning and reasoning about actions and change through diverse test cases for large language models.The suite is designed as an extensible benchmark for evaluating current and future LLMs.Source: §8 · p. 9

Findings

• LLMs show subpar performance even in simple common-sense planning domains.This finding motivates continued assessment of planning capabilities using systematic benchmarks.Source: §8 · p. 9

Future Work

• Future extensions could add partial-correctness metrics and support more International Planning Competition domains.These changes would broaden the benchmark’s evaluation scope.Source: §8 · p. 10
• The authors hope PlanBench encourages evaluation across different LLM models and models finetuned for planning tasks [2] [4] [28] [36] [24] [32] [9].The intended use spans general-purpose and task-specialized systems.Source: §8 · p. 10

A.1 Broader Impact

• A.1 Broader Impact: PlanBench contains no personally identifiable or privacy-related information, but performance on it alone cannot ensure safe or correct plans in external deployment.The benchmark is intended as a progress marker for LLM planning and reasoning, not as a standalone safety guarantee.Source: §A.1 · p. 14

Experimental setup

• Experiments used OpenAI API defaults with temperature 0; GPT-4 had an 8k context window and was evaluated from March to June, while Fast Downward and VAL handled planning and validation.The temperature setting made the LLMs deterministic.Source: §A.2 · p. 14

Obfuscation effects

• GPT-4 and Instruct-GPT3 both showed significant performance decreases when the domain was obfuscated deceptively or randomly.The comparison is reported for the Plan Generation test case in Table 2.Source: §A.3 · p. 14, Table 2 · p. 15
• The finding applies to both deceptive and random domain obfuscation in the evaluated plan-generation setting.Table 2 reports results for GPT-4 and Instruct-GPT3 on the Plan Generation test case.Source: §A.3 · p. 14, Table 2 · p. 15

Interpretation

• The degradation under obfuscation indicates that the models’ Blocksworld planning performance was more likely driven by pattern matching than reasoning.Reasoning should have remained robust to these kinds of obfuscation.Source: §A.3 · pp. 14–15

A.4.1 Comparison to the dataset

• Figure 4 compares the distribution of all Blocksworld instances by optimal plan length with GPT-4’s correct plans across those lengths.The visual directly contrasts dataset instances with GPT-4 successes.Source: Figure 4 · p. 15
• Thus, GPT-4’s correct-plan distribution does not simply favor Blocksworld instances with shorter optimal plans.The observed distribution differs from the intuition that shallower instances should be easier for planners.Source: Figure 4 · p. 15, §A.4.1 · p. 15
• The comparison also examines block-count distributions for instances where GPT-4 provides a correct plan versus all instances.Both optimal plan length and block count are included in the dataset comparison.Source: §A.4.1 · p. 15
• Smaller optimal plan lengths do not necessarily correspond to higher GPT-4 performance.The analysis characterizes these smaller-plan instances as more “shallow.”Source: §A.4.1 · p. 15
• This behavior may seem surprising relative to classical planners, for which shallower instances might be expected to support better performance.The passage explicitly contrasts the observation with classical-planner intuition.Source: §A.4.1 · p. 15
• The analysis attributes the pattern to LLM behavior: GPT-4 predicts the next token from the instance-specific context and learned weights.Consequently, instance shallowness alone need not determine whether the generated plan is correct.Source: §A.4.1 · p. 15

A.4.2 Failure modes analysis

• The analysis ignores either action delete conditions, all action preconditions, or both to provide a more forgiving validity assessment.Precondition relaxation treats every action as executable in any state, while delete relaxation ignores delete conditions.Source: §A.4.2 · p. 15
• Because correctness in the unrelaxed model implies correctness in every relaxed model, failures after relaxation identify plans invalid even under weaker requirements.Source: §A.4.2 · p. 15
• Even under the most relaxed evaluation, GPT-4 still fails on many plan-generation instances.This result evaluates generated plans against delete-relaxed, precondition-relaxed, and jointly relaxed domain models.Source: §A.4.2 · p. 15, Figure 5 · p. 16

A.5.1 Blocksworld

• In Blocksworld, the prompt defines pickup, unstacking, putting down, and stacking actions together with hand, clearance, and support restrictions.The task examples provide initial block configurations and goals, including placing the orange block on top of the blue or red block.Source: §A.5.1 · p. 16
• A Blocksworld example is marked SUCCESS when the extracted plan matches the four-action ground-truth solution after removing two unnecessary actions.The extracted plan includes unstacking and putting down d and a, then picking up c and stacking c on a; the ground truth omits the unnecessary actions involving a.Source: §A.5.1 · p. 17

A.5.2 Logistics

• Logistics prompts model package transport within cities by trucks and between cities by airplanes, using loading, unloading, driving, and flying actions.The domain provides one truck per city, an airport location in each city, and preconditions governing co-location and movement.Source: §A.5.2 · p. 17
• The Logistics ground-truth solution transports package_0 from city_1 to city_0 by loading it into airplane_1, flying, and unloading.The listed sequence is (load-airplane p0 a1 l1-0), (fly-airplane a1 l1-0 l0-0), and (unload-airplane p0 a1 l0-0).Source: §A.5.2 · pp. 17–18

A.5.3 Mystery Blocksworld

• Mystery Blocksworld replaces ordinary block names with abstract actions and predicates whose preconditions and effects describe changing relations among objects.Its actions are Attack, Feast, Succumb, and Overcome, with effects involving Pain, Province, Planet, Harmony, and Craves relations.Source: §A.5.3 · p. 18
• The Mystery Blocksworld examples target changing Craves relations, including goals for object c to crave object b or object a.One supplied ground-truth plan is (feast object d object c), (succumb object d), (attack object c), (overcome object c object a).Source: §A.5.3 · pp. 18–19

A.6.1 Blocksworld

• The Blocksworld prompt models one-minute pickup, unstack, put-down, and stack actions under hand, clearance, support, and table constraints.These rules define when blocks can be manipulated and how stacking changes clearance and hand occupancy.Source: §A.6.1 · p. 19
• A 4-minute Blocksworld plan achieves the goal by unstacking blue, putting it down, picking up orange, and stacking orange on blue.The prompt assigns one minute to each listed action and requires minimizing execution time.Source: §A.6.1 · p. 19
• A 6-minute GPT-4 Blocksworld response includes two unnecessary unstack-and-put-down actions, whereas the extracted and ground-truth plans omit them.The response removes yellow from orange and red from blue before completing the goal; the ground-truth sequence uses only four actions.Source: §A.6.1 · pp. 19–20

A.6.2 Logistics

• In Logistics, the prompt plans package transport between cities using trucks and airplanes, with loading and unloading taking 1 minute, driving 2 minutes, and flying 5 minutes.The domain constrains loading by co-location and restricts trucks to locations within a city while connecting cities through airports.Source: §A.6.2 · p. 20
• A Logistics example gives a 4-minute plan that flies airplane_1 to package_0, loads it, returns, and unloads it at location_0_0.A second instance supplies the three-action ground-truth sequence: load, fly, and unload.Source: §A.6.2 · pp. 20–21

A.6.3 Mystery Blocksworld

• Mystery Blocksworld replaces familiar block actions with Attack, Feast, Succumb, and Overcome, each taking 1 minute and changing symbolic facts.The prompt defines preconditions and effects involving Province, Planet, Pain, Harmony, and craving relations.Source: §A.6.3 · p. 21
• A Mystery Blocksworld example reaches the craving goal with a 4-minute sequence: Feast, Succumb, Attack, and Overcome.The second instance provides a ground-truth sequence of Feast, Succumb, Attack, and Overcome for a different craving goal.Source: §A.6.3 · p. 21

A.7.1 Blocksworld

• In Blocksworld, GPT-4 correctly verifies a valid plan that unstacked blue, put it down, and stacked orange on red.The verification states that the plan is valid.Source: §A.7.1 · p. 22
• In Blocksworld, GPT-4 identifies invalid plans by reporting either an unmet goal condition or unmet action preconditions.One failed plan leaves orange not on white; another cannot pick up yellow because it is not on the table, despite being clear with an empty hand.Source: §A.7.1 · pp. 22–23
• A Blocksworld verification matches ground truth when GPT-4 rejects a plan because the orange block remains off the red block.The example marks this agreement as SUCCESS.Source: §A.7.1 · p. 23

A.7.2 Logistics

• In Logistics, the cited plan is verified as valid when package_1 is loaded into airplane_0 at location_1_0, flown to location_0_0, and unloaded there.Source: §A.7.2 · p. 24
• Logistics verification catches invalid plans through both unmet action preconditions and unmet goal conditions.Examples identify package_3 not being in truck_1, package_1 not reaching location_2_0, and airplane_1 not being at location_0_0 before unloading.Source: §A.7.2 · pp. 24–25

A.7.3 Mystery Blocksworld

• The Mystery Blocksworld prompt renames actions and predicates while specifying their preconditions and the facts each action makes true or false.Its example starts with object b craving object a and seeks a plan achieving that object c craves object a.Source: §A.7.3 · p. 25
• Mystery Blocksworld verification prompts classify plans as valid or invalid, identifying unmet goal conditions or action preconditions when verification fails.One invalid plan fails because object c craves object e is unmet, while another has unmet preconditions for attack object d: harmony, planet object d, and province object d.Source: §A.7.3 · p. 26

A.8.1 Blocksworld

• Blocksworld execution prompts specify pickup, unstacking, placement, and stacking constraints, then ask for the resulting state after action sequences.The constraints include clear-block and empty-hand requirements, while execution examples track block positions and whether the hand is holding a block.Source: §A.8.1 · p. 27
• In one Blocksworld completion, GPT-4’s resulting-state response describes the yellow block as held and the orange block as clear, alongside the remaining stack relations.The prompt also reports an extracted LLM plan and a separate ground-truth plan for comparison.Source: §A.8.1 · p. 27

A.8.2 Logistics

• Logistics execution prompts model loading, unloading, truck driving, and airplane flights under location, city, and transport-state preconditions.An example tracks airplane_1 returning to location_0_0 with package_0 still in the airplane, while another supplies a ground-truth state containing in_p0_a1.Source: §A.8.2 · pp. 28–29

A.8.3 Mystery Blocksworld

• Mystery Blocksworld execution prompts define action effects for attack, succumb, overcome, and feast, including facts that become true or false.The examples provide resulting-state sequences and a ground-truth plan represented with predicates such as province_c, pain_d, and craves_a_b.Source: §A.8.3 · pp. 29–30

A.9.1 Blocksworld

• The replanning Blocksworld prompt uses the same stacking action vocabulary and restrictions to frame a separate replanning task.The supplied excerpt establishes the domain actions and clear-block, table, and hand-state constraints but does not provide a replanning result.Source: §A.9.1 · p. 30
• After an unexpected Blocksworld state change, the replanned solution unstacked the orange block from the red block and stacked it on the blue block.Source: §A.9.1 · p. 30
• In a second Blocksworld case, GPT-4 reported that no further actions were needed because the unexpected state already satisfied the goal.Source: §A.9.1 · p. 31

A.9.2 Logistics

• The Logistics prompt defines truck transport within cities, airplane transport between cities, package loading and unloading, and state-changing action restrictions.Source: §A.9.2 · p. 31
• Logistics replanning restored package_0 transport by flying airplane_1 to its origin, loading the package, returning, and unloading it at the destination.Source: §A.9.2 · p. 32

A.9.3 Mystery Blocksworld

• After an unexpected Mystery Blocksworld transition, the replanned actions were to feast object c from object a and overcome object c from object b.Source: §A.9.3 · p. 33

A.10.1 Blocksworld

• The Blocksworld generalization prompt requires arranging eight blocks into specified stacks under hand, clearance, pickup, unstacking, placement, and stacking constraints.Source: §A.10.1 · p. 33
• In Blocksworld, the GPT-4 completion produces a plan whose actions are decomposed into pickups and stacks, while the ground-truth plan contains the required cyan-green and silver-cyan construction.The completion also includes blue-silver, red-blue, black-red, and violet-black stacking actions, matching the remaining ground-truth steps shown separately.Source: §A.10.1 · pp. 34–35

A.10.2 Logistics

• The Logistics prompt defines truck loading, unloading, driving, and airplane loading, unloading, and flying, with preconditions and state updates governing package and vehicle locations.Trucks move only within a city, airplanes connect city airports, and loading removes packages from locations while unloading restores them at the vehicle’s location.Source: §A.10.2 · p. 35
• The Logistics example supplies a multi-package goal and an extended plan coordinating local truck routes with intercity airplane transfers.The plan routes packages among locations in three cities, including truck deliveries within cities and airplane transfers between airport locations.Source: §A.10.2 · pp. 35–36

A.10.3 Mystery Blocksworld

• Mystery Blocksworld replaces familiar action names with abstract actions whose preconditions and effects encode state changes such as Province, Planet, Pain, Harmony, and Craves relations.Attack, Succumb, Overcome, and Feast respectively alter these predicates through specified positive and negative effects.Source: §A.10.3 · pp. 37–38
• For one Mystery Blocksworld instance, GPT-4 generates attack-and-overcome pairs that construct the requested Craves chain, while a second instance provides a distinct ground-truth sequence.The first completion follows h, b, d, g, i, f, and j; the second ground-truth plan follows h, k, b, a, g, and j.Source: §A.10.3 · p. 38

A.11.1 Blocksworld

• The Blocksworld plan-reuse prompt requires manipulating clear blocks with an empty hand, and its completion moves the yellow block aside before placing orange on red.The stated constraints govern picking up, unstacking, putting down, and stacking, including how these actions change hand occupancy and block clearance.Source: §A.11.1 · p. 39
• The reuse example’s explicit plan is: unstack yellow from orange, put yellow down, pick up orange, and stack orange on red.This sequence achieves the stated goal that the orange block is on top of the red block.Source: §A.11.1 · p. 39
• The Blocksworld example requires moving the yellow block off the orange block and the red block off the blue block, with the ground-truth plan containing only the yellow-block moves.GPT-4’s extracted plan additionally unstacked and put down the red block.Source: §A.11.1 · p. 39

A.11.2 Logistics

• The Logistics prompts define truck and airplane transport with loading, unloading, driving, flying, and state-transition restrictions for package movement between cities.The examples include a plan loading package_0 into airplane_1, flying it from city_1 to city_0, and unloading it at the destination.Source: §A.11.2 · p. 40
• A second Logistics instance gives a ground-truth plan that loads package_0 into airplane_1 and flies the airplane from location_1_0 to location_0_0.Its goal state places airplane_1 at location_0_0 with package_0 inside it.Source: §A.11.2 · p. 40

A.11.3 Mystery Blocksworld

• Mystery Blocksworld renames domain actions and encodes their preconditions and effects, including Attack, Succumb, Overcome, and Feast.The supplied rules describe facts becoming true or false after each action, such as Pain following Attack and Harmony following Overcome.Source: §A.11.3 · p. 41
• The Mystery Blocksworld example reaches the goal that object c craves object a through Feast, Succumb, Attack, and Overcome actions.A separate ground-truth example uses Feast followed by Succumb to establish a different goal state.Source: §A.11.3 · p. 41

A.12.1 Blocksworld

• The reformulated Blocksworld prompt shuffles goal predicates while preserving action constraints such as clear-block, hand-empty, holding, and stacking requirements.The displayed ground-truth plan unstacked and put down the yellow block, then picked up and stacked the orange block on the red block.Source: §A.12.1 · p. 42

A.12.2 Logistics

• The reformulated Logistics prompt likewise retains package, truck, airplane, city, and action restrictions while presenting the transport task with reordered goal predicates.The supplied passage specifies loading and unloading conditions and restricts truck movement to locations within the same city.Source: §A.12.2 · pp. 42–43
• The Logistics examples require transporting package_0 between locations by loading it into airplane_1, flying between cities, and unloading it at the destination.The corresponding ground-truth sequence is load, fly, and unload.Source: §A.12.2 · p. 43

A.12.3 Mystery Blocksworld

• Mystery Blocksworld’s reformulated-goal example uses Feast, Succumb, Attack, and Overcome actions to achieve object c craves object a.The ground-truth plan is feast object d from object c, succumb object d, attack object c, and overcome object c from object a.Source: §A.12.3 · p. 44

A.13.1 Blocksworld

• The Blocksworld full-to-partial prompt defines pickup, unstack, put-down, and stacking constraints over block clearance, hand occupancy, and support relations.Its stated goal includes placing the orange block on top of the red block.Source: §A.13.1 · pp. 44–45
• GPT-4’s Blocksworld completion includes extra unstacking and put-down steps, whereas the supplied ground-truth plan directly proceeds from clearing the yellow block to stacking orange on red.The completion and ground truth are both explicitly shown in the prompt.Source: §A.13.1 · p. 45

A.13.2 Logistics

• The Logistics full-to-partial prompt formalizes package transport within cities by trucks and between cities by airplanes, with direct intra-city connections, airport locations, and location-based loading restrictions.Source: §A.13.2 · pp. 45–46
• The Logistics reformulation’s ground-truth plan transports package_0 from location_1_0 to location_0_0 using airplane_1.It consists of loading the package, flying the airplane, and unloading the package.Source: §A.13.2 · p. 46

A.13.3 Mystery Blocksworld

• The Full-to-Partial Mystery Blocksworld prompt introduces Attack, Feast, Succumb, and Overcome actions with preconditions and state changes.The supplied continuation explicitly specifies Attack’s required facts and its resulting true and false facts.Source: §A.13.3 · pp. 46–47
• A second Mystery Blocksworld example presents the same four-step plan as a proposed plan and as ground truth for the goal object c craves object a.Its initial state contains two Craves relations, harmony, two Planet facts, and two Province facts, while the goal contains the new craving relation.Source: §A.13.3 · p. 47

A.14.1 Blocksworld

• The Blocksworld prompt defines four actions—pick up, unstack, put down, and stack—under hand, clearance, table, and holding restrictions that determine their applicability and effects.Picking up or unstacking leaves the agent holding the block; putting down or stacking empties the hand, and stacking makes the supporting block non-clear.Source: §A.14.1 · pp. 47–48
• Blocksworld’s reformulated goal requires orange on red while preserving the red-blue stack and placing yellow on the table, with the ground-truth plan unstacking yellow, putting it down, then stacking orange on red.The corresponding initial state has red on blue, yellow on orange, both lower blocks on the table, and an empty hand.Source: §A.14.1 · p. 48

A.14.2 Logistics

• The Logistics prompt models package transport with truck and airplane loading, unloading, driving, and flying, constrained by shared locations, city connectivity, and vehicle-location effects.Trucks travel between locations within a city, whereas airplanes fly between city airports; loading removes a package from its location and unloading restores it at the vehicle’s location.Source: §A.14.2 · pp. 48–49
• Logistics reformulation moves package_0 from location_1_0 to location_0_0 by loading it into airplane_1, flying between cities, and unloading it at the destination.The full reformulated goal also specifies airplane_1 at location_0_0 while retaining the stated airport, truck, package, and city-location facts.Source: §A.14.2 · p. 49

A.14.3 Mystery Blocksworld

• Mystery Blocksworld encodes opaque action schemas whose preconditions and effects transform Province, Planet, Pain, Harmony, Object, and Craves relations.Attack creates Pain while removing Province and Planet; Succumb removes Pain; Overcome and Feast respectively update province/object relations and craving facts.Source: §A.14.3 · pp. 49–50
• Mystery Blocksworld’s reformulated-goal instance has the ground-truth plan: feast object d from object c, succumb object d, attack object c, and overcome object c from object a.The example’s initial state includes object a craves object b, object d craves object c, harmony, planets b and c, and provinces a and d; the reformulated goal additionally requires object c craves object a, planet object d, and province object c.Source: §A.14.3 · p. 50