On the Planning Abilities of Large Language Models : A Critical Investigation

• The study asks how effective LLMs are at standalone commonsense planning and how useful they are as heuristic guidance for other planning agents.These questions distinguish generating guaranteed-correct plans from supplying plausible suggestions to sound planners or humans in the loop.Source: §1 · p. 2
• The authors evaluate both modes systematically on generated instances modeled after International Planning Competition domains using automated planning tools.This automation avoids the subjectivity of earlier anecdotal evaluations of LLM reasoning capabilities.Source: §1 · p. 2
• GPT-4 achieves only about 12% executable, goal-reaching plans autonomously, with model choice and fine-tuning having little effect; obfuscation worsens performance.Human performance is substantially better in the simplest domain, and relaxed executability conditions further analyze GPT-4’s limitations.Source: §1 · p. 2
• In the LLM-Modulo setting, LLM-generated plans help the LPG planner search for correct plans more efficiently than empty- or random-plan seeds.LPG repairs candidate plans using a correct domain model, and the LLM+LPG combination requires fewer average search steps than both baselines.Source: §1 · p. 3
• External verification with VAL improves plan correctness by identifying errors and enabling repeated back-prompting of the LLM.The verifier provides feedback on generated plans, which the LLM uses to produce improved plans in commonsense domains.Source: §1 · p. 3

Evaluation setup

• Explicit domain prompts constrain LLMs to listed actions, enabling zero-shot and few-shot evaluation against executable plans rather than plausible commonsense plans.This simplifies approaches such as SayCan [2], which filters or interprets plans using executor-available skills and internal model probabilities.Source: §2 · p. 3

Evaluation modes

• Interactive evaluations risk a Clever Hans effect, while translation-focused systems [38] [21] do not reveal LLMs’ internal planning capabilities.The paper therefore separates autonomous evaluation from assistance to external planners and reasoners; parallel studies [21] largely corroborate the difficulty of finding executable plans.Source: §2 · p. 3

Planning decomposition

• LLMs provide broad action knowledge but struggle to sequence known actions into valid, interaction-free causal plans, making them complementary to standard AI planners.LLMs can support action discovery through candidate plans or domain models, while planners focus on causal chaining and conflict-free sequencing.Source: §2 · p. 3

3.1 Background

• Classical planning specifies a domain, initial state, and goal state, with actions defined by preconditions and add/delete effects; a plan is an action sequence reaching the goal.PDDL is the standard representation, illustrated with a Blocksworld pickup action.Source: §3.1 · p. 4
• The section motivates the prompt experiments as tests of LLM reasoning about actions and state changes in goal-directed deterministic planning problems.Classical planning state spaces are defined by possible truth assignments over predicates.Source: §3.1 · p. 4

3.2 Prompt Generation

• The generation pipeline converts PDDL domain files into experiment-specific prompts by creating problems, optionally adding valid planner-generated examples, and translating natural-language variants.A domain-specific translator handles the final conversion from PDDL for all natural-language experiments.Source: Figure 2 · p. 4, §3.2 · p. 5
• The prompt suite crosses natural-language versus PDDL presentation with zero-shot versus one-shot examples, producing four experimental configurations.Natural-language prompts describe actions, states, goals, and plans as complete sentences, whereas PDDL prompts encode all elements in PDDL.Source: §3.2 · pp. 4–5
• Prompts provide a lifted domain description and end with a planning problem; one-shot prompts additionally include an example instance, valid plan, and plan-end tag for extraction.Outputs that cannot be reasonably parsed into an instance are marked incorrect.Source: §3.2 · p. 4
• A fifth experiment adds state-tracking chain-of-thought prompting to the natural-language one-shot setting, annotating action states, applicability reasons, and resulting states.The example also includes a meta-explanation of plan correctness, and the LLM is asked to reproduce the annotations.Source: §3.2 · p. 5
• Example generation is modular: although the setup uses a planner, any method producing valid plans can supply one-shot examples.The pipeline separately adds justification annotations in the state-tracking experiment.Source: §3.2 · p. 5

Evaluation setup

• The autonomous evaluation translates LLM-generated natural-language plans into validator-compatible forms and checks them with VAL across deterministic GPT-family experiments.GPT-4 and GPT-3.5 were tested across prompt configurations, whereas Instruct-GPT3 and GPT3 were tested on one-shot natural-language prompts.Source: §4 · pp. 5–6

Blocksworld

• GPT-4 improved over earlier GPT models across Blocksworld prompts but solved only approximately 34% overall, with chain-of-thought prompting providing little benefit.Natural-language prompts outperformed PDDL prompts, and replacing an Instruct-GPT3 example sharply reduced accuracy, consistent with pattern matching rather than induced planning models.Source: §4 · p. 6, Table 2 · p. 6

Model variants

• Fine-tuned GPT-3 solved 122 of 600 Blocksworld instances, approximately 20%, while BLOOM solved 1.6% in Blocksworld and none in Mystery Blocksworld.The fine-tuning dataset contained 1,000 separate instances; BLOOM was evaluated with one-shot natural-language prompts.Source: §4 · pp. 6–7

Human Baseline

• Humans solved 78% of Blocksworld instances validly and 89.7% of their valid plans optimally, contrasting with the subpar autonomous performance of LLMs.The study involved 50 participants and used instances drawn from the 600-instance evaluation set.Source: §4 · p. 7

Logistics

• GPT-4 solved 14% (28/200) of Logistics instances one-shot and 7.5% (15/200) zero-shot with natural-language prompts, while GPT-3.5 performed worse.With PDDL prompts, GPT-4 achieved 14% (28/200) one-shot and 5.5% (11/200) zero-shot.Source: Table 2 · p. 6, §4 · p. 7

Obfuscation

• Obfuscating Blocksworld names reduced GPT-4 from 210 solved zero-shot natural-language instances to 1 deceptive and 0 randomized instances, indicating brittleness to surface patterns.For PDDL prompts, GPT-4 decreased from 106 solved Blocksworld instances to 3 deceptive Mystery Blocksworld instances; chain-of-thought prompting did not significantly help.Source: Table 2 · p. 6, §4 · p. 7

Analyzing GPT-4 failures

• Under the most lenient delete-and-precondition relaxation, approximately 39% of GPT-4 plans remained incorrect because they failed to reach the goals.Relaxations increase apparent correctness, while precondition relaxation makes every plan executable; remaining failures can therefore still reflect unmet goals.Source: §4 · p. 8, Figure 3 · p. 8

5.1 LLM Plans as Heuristics to Sound Planners

• GPT-4 seed plans significantly reduced LPG search steps in Blocksworld and Logistics4 versus empty and random seeds, but offered little benefit or a penalty in Mystery Blocksworld.Across domains, edit distances were approximately half the seed-plan length, indicating that about 50% of final plans retained elements of the LLM suggestions.Source: §5.1 · pp. 8–9

5.2 Verifier-assisted repeated backprompting of LLMs

• 82% of Blocksworld and 70% of Logistics failures became correct within 15 VAL [11] backprompting rounds, whereas Mystery Blocksworld performance remained poor.Successful cases required 3.68 average rounds in Blocksworld and 3.31 in Logistics; the Logistics evaluation was restricted to instances solvable within 550 LPG search steps.Source: §5.2 · pp. 9–10, Table 4 · p. 9

5.3 LLMs as idea generators for humans-in-the-loop

• In a 49-versus-48 participant study, LLM-generated suggestions produced no significant differences in accuracy, time, or cognitive load, while 3 assisted participants accepted incorrect suggestions.Two participants submitted erroneous LLM suggestions as their plans, demonstrating potential automation bias.Source: §5.3 · p. 10

Conclusion

• LLMs fail as autonomous planners, with GPT-3 performing dismally and GPT-4 still failing on mystery blocksworld despite improvement on blocksworld.The mystery blocksworld result indicates difficulty with abstract reasoning.Source: §6 · p. 10
• In the LLM-Modulo setting, LLM-generated plans improve search by sound planners such as LPG.Source: §6 · p. 10
• External verifiers identify plan errors and enable back-prompting, producing better plans in commonsense domains.Source: §6 · p. 10

A.1 Classical Planning Problem Formulation

• A classical planning problem is modeled as P = ⟨D, I, G⟩, where D is the domain, I the initial state, and G the goal specification.Possible truth assignments over predicates define the planning state space.Source: §A.1 · p. 14
• The domain D = ⟨F, O⟩ comprises fluents defining state variables and actions that can be performed in the planning problem.Each fluent corresponds to a predicate with some arity.Source: §A.1 · p. 14
• Each action has preconditions that must hold for execution and effects that specify predicates added to or deleted from the state.Effects are represented as ⟨add[V], del[V]⟩.Source: §A.1 · p. 14
• Actions are grounded when variables are replaced by objects; otherwise, the domain model remains lifted.The same lifted-versus-grounded distinction applies to predicates.Source: §A.1 · p. 14
• In Blocksworld, picking up a block requires it to be clear and on the table with an empty arm, then makes it held and removes those conditions.The PDDL action also makes the arm non-empty.Source: §A.1 · p. 15
• A plan is an action sequence that reaches a state satisfying the goal from the initial state, while an optimal plan has no lower-cost alternative.Actions may be associated with costs.Source: §A.1 · p. 15
• The formalism can be extended with object typing, richer preconditions, conditional effects, and broader planning formalisms.These extensions go beyond the simpler class of models described here.Source: §A.1 · p. 15

Comparisons between the instances and plans generated by GPT-4

• The study compared Blocksworld and Logistics instance distributions by optimal plan length with GPT-4’s distribution of correct plans.This comparison used Figures 4–7.Source: §A.2 · p. 15
• Figures 4–7 indicate that traditional notions of planning complexity do not hold for LLM-generated plans.An instance’s optimal-plan complexity does not reliably predict GPT-4’s ability to generate a correct plan.Source: §A.2 · p. 15
• Figure 4 compares Blocksworld instances with those for which GPT-4 generated correct plans using PDDL or natural-language prompts containing one example.The figure covers both prompting styles.Source: Figure 4 · p. 15

A.3 Failure modes

• Relaxation increases goal-reaching GPT-4 plans in Mystery Blocksworld, yet many plans remain non-goal-reaching even under Delete+Precondition Relaxation.In precondition-relaxation modes, a plan is inexecutable when an action lacks the required number of parameters.Source: §A.3.1 · p. 15

A.3.1 LLM failures

• In relaxed Mystery Blocksworld assessments, inexecutable plans are identified by actions lacking the required number of parameters.Source: §A.3.1 · p. 15
• Across both Mystery Blocksworld and Logistics, increasing assessment relaxation improves goal-reaching outcomes without eliminating planning failures.Source: §A.3.1 · p. 15
• Further relaxation likewise increases goal-reaching GPT-4 plans in Logistics, but non-goal-reaching plans remain even in the most relaxed configuration.Source: §A.3.1 · p. 15
• Figure 5 compares Blocksworld instances with instances where GPT-4 generated correct plans using example-free PDDL or natural-language prompts.Source: Figure 5 · p. 16
• Figure 6 compares Logistics instances with instances where GPT-4 generated correct plans using PDDL or natural-language prompts containing one example.Source: Figure 6 · p. 16

A.3.2 Human failures

• 11 of 50 human-baseline participants failed to produce a valid plan, and all 11 produced inexecutable plans.Source: §A.3.2 · p. 16
• 10 of 49 participants without LLM assistance failed to produce a valid plan, and all 10 produced inexecutable plans.Source: §A.3.2 · p. 16
• 15 of 48 participants with LLM assistance failed to produce a valid plan; 14 produced inexecutable plans and 1 produced a non-goal-reaching plan.Source: §A.3.2 · p. 16

A.4.1 Domain description

• Blocksworld permits picking up, unstacking, putting down, and stacking blocks, subject to hand, clearance, table, holding, and stacking preconditions.Actions update whether blocks are clear and whether the hand is empty or holding a block.Source: §A.4.1 · pp. 16–17

A.4.2 One-shot prompt with GPT-4 plan

• A second one-shot example presents initial conditions and a goal requiring orange to be placed on red.The prompt demonstrates the same statement-goal-plan format for a different configuration.Source: §A.4.2 · p. 18
• The one-shot GPT-4 example matches the ground-truth plan for placing orange on red after removing yellow, and is marked successful.The extracted response includes additional red-blue unstacking and put-down actions absent from the ground-truth plan.Source: §A.4.2 · p. 18

A.4.3 Zero-shot prompt with GPT-4 plan

• The zero-shot GPT-4 response gives the four actions needed to move orange from below blue to on top of blue, matching the listed ground-truth plan and receiving a success marker.The prompt asks only for the actions in the plan, without examples.Source: §A.4.3 · pp. 18–19

A.4.4 State-tracking prompt with GPT-4 plan

• State-tracking prompting defines correctness by sequentially checking action preconditions, applying effects, and verifying that the final state satisfies all goals.The example tracks unstack, put-down, pick-up, and stack operations through their resulting states.Source: §A.4.4 · p. 19
• The first state-tracking example reaches a final state with orange on blue, satisfies the goal conditions, and is declared valid.GPT-4 explicitly records the current state, action, reason, resulting state, and final state.Source: §A.4.4 · p. 19
• A second state-tracking example begins with red on blue and yellow on orange, then explicitly reasons through unstacking and putting down both obstructing blocks.The subsequent plan proceeds toward placing orange on red by updating each intermediate state.Source: §A.4.4 · p. 20
• The prompt records a current state in which the hand holds the orange block while the red, blue, and yellow blocks are clear and on the table.This state representation provides the conditions used to assess the next action.Source: §A.4.4 · p. 20
• It explains that stacking orange on red is applicable because the red block is clear and the hand is holding orange.The action’s stated preconditions are satisfied in the current state.Source: §A.4.4 · p. 20
• After stacking, the resulting state records orange on red, an empty hand, and the remaining blocks on the table.The final state preserves these conditions and is reported as satisfying the goal.Source: §A.4.4 · p. 20
• The prompt’s GPT-4 plan successfully satisfies the goal conditions and is marked as a success.The final state is explicitly judged valid, with the ground-truth plan reached after extraction.Source: §A.4.4 · pp. 20–21
• The extracted plan contains six actions, including two redundant unstack-and-put-down pairs before the final stacking sequence.The extracted sequence is (unstack red blue), (put-down red), (unstack yellow orange), (put-down yellow), (pick-up orange), and (stack orange red).Source: §A.4.4 · p. 21
• The ground-truth plan consists of four actions: unstack yellow orange, put-down yellow, pick-up orange, and stack orange red.The extracted plan’s final four actions match this ground-truth sequence.Source: §A.4.4 · p. 21

A.5.1 Domain description (Deceptive Disguising)

• The deceptively disguised domain presents renamed actions—Attack, Feast, Succumb, and Overcome—and specifies their preconditions plus add and delete effects.Examples include Attack producing Pain while deleting Province and Planet, and Overcome producing Harmony and Province while deleting other facts.Source: §A.5.1 · p. 21

A.5.2 One-shot prompt with GPT-4 plan (Deceptive Disguising)

• GPT-4’s one-shot plan fails against the stated ground-truth plan in the deceptive-disguising domain.The generated sequence is listed separately from the ground-truth sequence and marked “FAILURE.”Source: §A.5.2 · pp. 21–22

A.5.3 Zero-shot prompt with GPT-4 plan (Deceptive Disguising)

• GPT-4’s zero-shot plan likewise fails to match the ground-truth plan for a deceptive-disguising instance.The response supplies five extracted actions, while the ground-truth sequence is different and marked “FAILURE.”Source: §A.5.3 · p. 22

A.5.4 State-tracking prompt with GPT-4 plan

• Plan correctness is evaluated by checking sequential action executability through state transitions and verifying that the final state satisfies all goals.Each action must have its preconditions satisfied in the state produced by preceding actions, with add and delete effects applied.Source: §A.5.4 · p. 22
• With state tracking, GPT-4 produces a valid plan whose final state satisfies the goal conditions for the first demonstrated instance.The prompt explicitly records current states, action preconditions, resulting states, and the final goal-satisfying state.Source: §A.5.4 · pp. 23–24
• State tracking also yields a valid six-action extracted plan for a second instance, although the displayed ground-truth plan is marked as a failure.The plan proceeds through tracked attacks, an overcome action, and a final state satisfying the goal before the extracted and ground-truth plans are shown.Source: §A.5.4 · p. 24

A.5.5 Domain description (Randomized Disguising)

• A randomized-disguising variant replaces action and predicate names with opaque alphanumeric strings while retaining stated action restrictions and effects.The supplied example uses strings such as “1jpkithdyjmlikck” and “aqcjuuehivl8auwt” instead of interpretable names.Source: §A.5.5 · p. 24
• The randomized domain disguises actions and predicates as opaque strings, while specifying their preconditions and effects in natural-language templates.Examples state what must be true before actions and what becomes true or false afterward, using randomized identifiers for actions, objects, and facts.Source: §A.5.5 · p. 24
• The randomized action descriptions specify both enabling conditions and state changes, including facts that become false after execution.This representation defines action semantics through paired natural-language precondition and effect statements over disguised predicates and objects.Source: §A.5.5 · pp. 24–25

A.5.6 One-shot prompt with GPT-4 plan (Randomized Disguising)

• In the one-shot setup, GPT-4 receives initial conditions and a goal before producing a natural-language action plan for extraction.The prompt asks for achieving a specified relation between objects and presents the requested plan in a one-shot format.Source: §A.5.6 · p. 25, §A.5.7 · p. 25
• The one-shot example’s plan contains four extracted actions, but all four differ in object assignments from the four-step ground truth.The generated actions use object b, object a, object d, and object d, while the ground truth uses object d, object d, object c, and object c.Source: §A.5.6 · p. 25
• GPT-4 fails the one-shot instance: its extracted plan differs from the ground-truth sequence.The response proposes actions involving objects a and b, whereas the ground truth requires actions involving objects d and c.Source: §A.5.6 · p. 25

A.5.7 Zero-shot prompt with GPT-4 plan (Randomized Disguising)

• In the zero-shot setup, GPT-4 is directly asked to provide only the actions needed to achieve the stated goal.The response is formatted as a numbered natural-language plan and then converted into extracted symbolic actions.Source: §A.5.7 · pp. 25–26
• GPT-4 also fails the zero-shot instance, returning an action sequence that does not match the ground-truth plan.The generated sequence uses object a and object c in different roles from the required ground-truth sequence involving object b and object c.Source: §A.5.7 · pp. 25–26

A.6.1 Domain description

• The domain transports packages within cities via trucks and between cities via airplanes, with one truck and one airport location per city.Locations within a city are directly connected, and cities are connected for airplane travel.Source: §A.6.1 · p. 26
• Available actions are loading and unloading packages, driving trucks between locations, and flying airplanes between cities.These actions define the transport operators used in both GPT-4 demonstrations.Source: §A.6.1 · p. 26
• Truck loading requires package-truck co-location, while loading removes the package from the location and places it in the truck.Truck unloading requires the package to be in the truck and restores it at the truck’s location.Source: §A.6.1 · p. 26
• Airplane loading likewise requires co-location and transfers the package into the airplane, while unloading requires onboard status and places it at the airplane’s location.Airplane movement is governed separately from truck movement and operates between city airports.Source: §A.6.1 · p. 26

A.6.2 One-shot prompt with GPT-4 plan

• The cited prompt text specifies initial conditions including package_0 at location_1_0 and states the goal of having package_0 at location_0_0.Source: §A.6.2 · p. 26, §A.6.3 · p. 27
• The successful ground-truth plan loads package_0 onto airplane_1 at l1-0, flies it to l0-0, and unloads package_0 there.Source: §A.6.2 · p. 27

A.6.3 Zero-shot prompt with GPT-4 plan

• Zero-shot GPT-4 planning fails: its extracted ten-action plan differs from the four-action ground-truth plan and is marked FAILURE.The generated plan includes invalid or unnecessary truck and airplane actions, whereas the ground-truth plan flies airplane_1, loads the package, returns, and unloads it.Source: §A.6.3 · p. 28

A.7.1 Domain description

• The Blocksworld appendix defines a four-operator STRIPS domain using predicates such as clear, ontable, handempty, holding, and on.The domain includes pickup, put-down, stack, and unstack-style preconditions and effects, and prompts request plans in PDDL syntax.Source: §A.7.1 · p. 28

A.7.2 One-shot prompt with GPT-4 plan

• GPT-4’s one-shot Blocksworld plan fails because it unstacked the wrong object, whereas the ground-truth plan requires unstacking d from c.The extracted plan was (unstack a b), (put-down a), (pick-up c), (stack c a), while the ground truth begins (unstack d c).Source: §A.7.2 · p. 29

A.7.3 Zero-shot prompt with GPT-4 plan

• GPT-4’s zero-shot Blocksworld response succeeds, exactly matching the ground-truth sequence of unstack, put-down, pick-up, and stack actions.The response and ground truth both use (unstack b c), (put-down b), (pick-up c), (stack c b).Source: §A.7.3 · p. 30

A.8.1 Domain description (Deceptive Disguising)

• The deceptively disguised domain replaces Blocksworld predicates with terms such as province, planet, harmony, pain, and craves while retaining analogous operator structure.Its one-shot and zero-shot prompts use the renamed mystery-4ops domain and require PDDL plan output.Source: §A.8.1 · p. 30

A.8.2/A.8.3 Deceptive-disguising prompts

• Both deceptive-disguising GPT-4 examples fail: the one-shot response chooses feast a b instead of feast d c, and the zero-shot response produces a different invalid sequence.The one-shot ground truth begins (feast d c), while the zero-shot ground truth is (feast b c), (succumb b), (attack c), (overcome c b).Source: §A.8.2 · p. 32, §A.8.3 · p. 32

A.9.1 Domain description

• The Logistics domain models objects, trucks, airplanes, locations, cities, and airports with load, unload, and transport actions.Its predicates include at, in, and in-city, while truck and airplane movement requires matching locations or airports.Source: §A.9.1 · pp. 32–34

A.9.2 One-shot prompt with GPT-4 plan

• GPT-4’s one-shot Logistics plan fails because it uses airplane a0 and an unnecessary return flight, whereas the ground truth loads and flies airplane a1 once.The response contains four actions, while the ground truth contains (load-airplane p0 a1 l1-0), (fly-airplane a1 l1-0 l0-0), and (unload-airplane p0 a1 l0-0).Source: §A.9.2 · p. 35

A.9.3 Zero-shot prompt with GPT-4 plan

• The zero-shot Logistics response gives a three-step natural-language plan: load p0 onto a0, fly a0 to l0-0, and unload p0.The supplied response presents the actions as LOAD-AIRPLANE, FLY-AIRPLANE, and UNLOAD-AIRPLANE rather than full PDDL syntax.Source: §A.9.3 · p. 36
• GPT-4’s zero-shot Airplane plan fails because its extracted sequence differs from the ground-truth plan.The extracted plan loads, flies, and unloads using airplane a0, whereas the ground truth uses airplane a1 and requires an additional flight.Source: §A.9.3 · p. 36

A.10.1 Blocksworld example with GPT-4

• The Blocksworld prompt specifies pickup, unstacking, placement, and stacking actions constrained by hand occupancy, block clearance, and stacking preconditions.These rules define when actions can be executed and how they change whether blocks and the hand are clear.Source: §A.10.1 · p. 36
• GPT-4 eventually produces a successful Blocksworld plan after validator feedback exposes unmet preconditions and goal conditions in successive attempts.Earlier plans repeatedly fail on blocks not being clear or on unmet goals, while the final plan is marked successful.Source: §A.10.1 · pp. 37–39

A.10.2 Mystery Blocksworld example with GPT-4

• The Mystery Blocksworld domain uses opaque action names such as Attack, Feast, Succumb, and Overcome, whose effects and preconditions determine whether plans can progress.The prompt describes state transitions including province, planet, pain, harmony, and craving relations.Source: §A.10.2 · p. 39
• In Mystery Blocksworld, GPT-4 repeatedly proposes invalid plans after validator feedback identifies unmet action preconditions, eventually determining that the target goal is impossible.The final conclusion concerns achieving object c craving object a under the supplied initial conditions and action restrictions.Source: §A.10.2 · pp. 40–41

A.10.3 Logistics example with GPT-4

• The Logistics example frames planning as transporting packages within cities by trucks and between cities by airplanes over connected locations and cities.Each city has exactly one truck and one airport location, and the prompt introduces the actions available for this setting.Source: §A.10.3 · p. 42
• GPT-4 ultimately produced a successful logistics plan after repeated verifier feedback identified unmet preconditions in earlier plans.The successful plan loads both packages at location_1_0, flies to location_0_0, and unloads them there.Source: §A.10.3 · pp. 42–43

A.11.1 LLM experiment details and the compute cost

• The GPT-4 experiments used deterministic temperature-0 API calls with an 8k context window; compute cost was $231 for autonomous experiments and $149 for back-prompting.The reported GPT-4 pricing was $0.03 per 1K prompt tokens and $0.06 per 1K completion tokens.Source: §A.11.1 · p. 44

A.11.2 LPG experiment details

• Heuristic experiments supplied LPG 1.2 with either an LLM plan, a same-length random plan, or no input plan, using local search, one restart, and up to 500 or 550 steps.LPG used its default heuristic evaluation function and omitted best-first fallback.Source: §A.11.2 · p. 44

A.12.2 Interface of the user study

• The study was conducted on Prolific, with participants paid $8.12/hour for the human baseline and $10.29/hour for the LLM+human study; interface images document study stages.The appendix includes interfaces for planning with and without LLM assistance, translation, and NASA-TLX assessment.Source: §A.12 · p. 44, Figure 12 · p. 48, Figure 13 · p. 48

A.12.1 Instructions provided to the participants

• The user studies asked participants to solve one Blocksworld problem by arranging blocks into stacks, with separate instructions for LLM-assisted, unaided, and human-baseline conditions.Participants received a 50-cent base bonus; incentives for success and failure varied by condition, and random submissions were rejected.Source: §A.12.1 · pp. 44–45

A.13 Broader Impact on using LLMs for planning

• LLM-generated plans pose safety risks because they may be factually incorrect, violate preconditions, fail to achieve goals, or produce detrimental side effects.The paper recommends verification through an automated verifier in heuristic mode or a human verifier in the loop.Source: §A.13 · pp. 45–46
• LLMs may perpetuate cultural or implicit biases in plans, but providing a carefully scrutinized domain model can reduce default-plan tendencies.The appendix illustrates this concern with a wedding-plan example and contrasts autonomous generation with domain-model-conditioned prompting.Source: §A.13 · p. 46