Planning Commitment of LLM under Partial Observability

1 Introduction

• Planning commitment is an understudied, uncontrolled choice: aggressive plans risk failed actions, while conservative planning requires more LLM calls.The question applies to both direct LLM planners and LLM formalizers that generate symbolic representations for an external solver.Source: §1 · p. 1
• The study compares aggressive and conservative planners and formalizers in two partially observable text-based domains: Coin Collector and ALFWorld.Coin Collector uses curated statistical priors; ALFWorld relies on commonsense knowledge from realistic room contents.Source: §1 · pp. 1–2
• Both strategies reach similar success rates, but conservative methods have far fewer failed attempts while aggressive methods are more efficient when priors or common sense are available.LLMs also treat provided priors nearly as facts, risking failure when priors are unreliable; hybrids with tunable hyperparameters control the safety–efficiency balance.Source: §1 · p. 2
• The paper treats planning commitment as a design axis independent of the planner–formalizer distinction and introduces an aggressive formalizer for a four-way comparison.Its contributions also include two environments, hybrid strategies with tunable hyperparameters, and recommendations for selecting commitment levels.Source: §1 · pp. 2–3

2 Related Work

• Conservative planners use closed-loop interaction, while aggressive planners generate long-horizon plans and repair them when execution deviates.The former can incur repeated inference cost and myopic behavior; the latter can have early modeling errors corrupt entire plans and often rely on implicit parametric knowledge.Source: §2 · p. 3
• Bounded-lookahead planners search short horizons before acting, an intermediate commitment level that the paper treats as outside its scope.Examples include LATS, LLM-guided Monte Carlo planning, and FLARE.Source: §2 · p. 3
• Conservative formalizers incrementally update symbolic models from observations, whereas uncertainty-aware planners generally require complete, hand-specified models.The cited LLM-related systems generate hypotheses, maintain distributions over sampled worlds, or learn symbolic POMDP models; prior work does not study how priors affect planning.Source: §2 · p. 3

Planning tasks

• A planning problem is a tuple of states, actions, transitions, initial state, and goal; the task is to find an action sequence that reaches the goal.The model receives natural-language domain and problem descriptions rather than the tuple itself; optional statistical priors describe hidden facts.Source: §3 · p. 3

Partial observability

• Under partial observability, some initial-state facts are withheld and become known only through actions and resulting observations or error messages.In Coin Collector, the model knows its current room and the coin's location but not room connections.Source: §3 · pp. 3–4

Interaction and success

• Each round supplies the task and interaction history, then executes a plan segment until it ends or an action is rejected; success is judged by simulator execution.A successful run may include rejected actions followed by replanning.Source: §3 · p. 4

3.1 Planning Methodologies

• The methodologies vary two independent choices: whether the LLM directly writes actions or formalizes a PDDL problem for a classical planner, and how much unobserved information it may use.Figure 2 summarizes the planning and formalization approaches.Source: §3.1 · p. 4, Figure 2 · p. 4
• Conservative agents use only described or observed facts, plan toward intermediate subgoals, and extend plans after successful segments.They ignore supplied priors, though LLM knowledge may still implicitly shape choices among observed options.Source: §3.1 · p. 4
• Aggressive agents fill missing facts using background knowledge or supplied priors, plan to the goal, and replan when actions are rejected.Any use of unobserved information counts as commitment; replanning continues from the current state rather than restarting.Source: §3.1 · p. 4
• The four combinations correspond to families of existing approaches, with execution semantics and methodology definitions provided in Appendix A.All approaches are evaluated using a simulator.Source: §3.1 · p. 4, Figure 2 · p. 4

4 Data

• Coin Collector is a 2D room-grid navigation domain with move and take actions, while ALFWorld represents household object placement tasks.Coin Collector has no inherent common sense, so its statistical priors serve as a controlled proxy.Source: §4 · pp. 4–5
• Coin Collector uses custom nine-room floor plans and 100 problems, each paired with fully and partially observable descriptions.Room connections are omitted in partial-observation descriptions; significant pair statistics are supplied as accurate priors or withheld in a no-prior setting.Source: §4 · pp. 4–5
• ALFWorld uses 100 generated source worlds to create partially observable five-room tasks with 15 receptacles, 15 movable objects, and one cross-room placement goal.Room connectivity and contents are omitted, and realistic room contents provide commonsense priors; there is no no-prior setting.Source: §4 · p. 5

5 Experimental Setup

• All methods receive the same domain description and initial observation; only aggressive Coin Collector methods receive statistical priors, while ALFWorld has none.Giving priors to conservative methods would itself constitute commitment, so the with-priors comparison contrasts the ends of the commitment spectrum.Source: §5 · p. 5
• Under partial observability, attempts are capped at k + 1 on Coin Collector and 20 on ALFWorld; formalizers may revise syntax errors up to three times.Here k is the number of hidden connections; plans are computed with metric-FF and executed in the authors’ simulators.Source: §5 · p. 5
• The evaluation reports planning accuracy and per-problem averages for failed attempts, API calls, and output tokens.API calls include every LLM call, including conservative plan extensions and PDDL revisions, and are the primary efficiency measure.Source: §5 · p. 5

Conservative vs. Aggressive Methods

• In fully observable environments, aggressive and conservative methods perform identically in most cases; the Conservative Formalizer drop comes from failing to update the agent’s location in PDDL.The authors attribute remaining behavior variation to prompt differences and model randomness, not commitment.Source: Conservative vs. Aggressive Methods · p. 6
• Without statistical priors, conservative methods have fewer failed attempts, while aggressive methods save only modest API calls; with priors, aggressive methods average around 40% fewer API calls than conservative methods.Without priors, aggressive methods must make assumptions that are usually incorrect, and their API-call counts are similar to conservative methods.Source: Conservative vs. Aggressive Methods · p. 6
• Aggressive planners always use more output tokens than conservative planners, while conservative formalizers use more tokens than aggressive formalizers.Source: Conservative vs. Aggressive Methods · p. 6
• On ALFWorld, conservative planners have fewer failed attempts, but aggressive planners use fewer API calls; aggressive formalizers also have fewer failed attempts and are more efficient than conservative formalizers.Formalizer methods have many failed attempts overall, likely reflecting domain difficulty; conservative methods use more output tokens on average.Source: Figure 3 · p. 6, Conservative vs. Aggressive Methods · p. 7
• Conservative methods use only observed connections, whereas aggressive methods often include unobserved connections; aggressive formalizers increasingly rely on higher-probability statistical priors.Figure 4 measures unobserved connections in problem files that are also present in priors for aggressive formalizers on Coin Collector.Source: Conservative vs. Aggressive Methods · p. 7, Figure 4 · p. 7
• Hybrid Planner (fixed) reduces failed attempts while keeping API calls close to Aggressive Planner; fixed hybrid formalizers for Gemini and Qwen also balance safety and efficiency.Figure 5 shows the tradeoff across evaluation metrics for partially observable Coin Collector, with Gemini results only.Source: Figure 5 · p. 8, Conservative vs. Aggressive Methods · p. 8
• Two hybrid formalizer cases have more failed attempts than both baselines: DeepSeek fixed and Gemini free; their errors involve room tracking and prior-derived hallucinated connections, respectively.DeepSeek’s incorrect start-room rate is higher than for Conservative Formalizer.Source: Conservative vs. Aggressive Methods · p. 8
• The recommended commitment level depends on API-call and failure costs: conservative for settings without priors, aggressive when failures are cheap and priors are reliable, and hybrid when both costs matter.The authors note that validating these recommendations in real-world settings remains future work.Source: Conservative vs. Aggressive Methods · p. 8

7 Conclusion

• The study compares aggressive and conservative LLM planners and formalizers in two controlled, partially observable environments.Source: §7 · p. 8
• Aggressive and conservative methods reach similar accuracy but trade safety against efficiency: aggressive methods save API calls when priors or common sense are available, while conservative methods generally fail less often.Without priors or common sense, aggressive methods save little while failing more often; conservative formalizers on ALFWorld are an exception to the lower-failure pattern.Source: §7 · p. 8
• Choose aggressive commitment when priors are reliable and failures are cheap to recover from, conservative commitment when priors are lacking or failures costly, and hybrid methods when both matter.Source: §7 · p. 8

8 Limitations

• Transfer beyond the two artificial text-based domains is untested, so the recommendations may not reflect real-world performance.The authors specifically name web navigation and lab automation as settings where transfer remains untested.Source: §8 · p. 9
• Priors come from the evaluation set and are perfectly calibrated; miscalibrated or held-out priors are not tested.The risk from incorrect priors is extrapolated from observed over-reliance, not measured directly.Source: §8 · p. 9
• Planning accuracy may produce false positives when a successful plan comes from PDDL that does not correctly describe the environment.The authors report no false positives in a 20-sample manual inspection.Source: §8 · p. 9
• Output-token counts underestimate generation cost because hidden reasoning tokens are excluded.Those tokens are not reported consistently across providers.Source: §8 · p. 9

A Formal Task and Methodology Formulation

• Plan validity is determined by online simulator execution, rather than offline comparison with ground-truth specification files.A run succeeds when its terminal state satisfies the goal, and may include rejected actions followed by replanning.Source: A Formal Task and Methodology… · p. 11, §A.1 · p. 12

A.1 Interactive Planning Environment

• A task instance includes domain and problem descriptions, valid action signatures, an initial observation, and optional common-sense priors.Under partial observability, the problem description omits some initial-state facts, which are revealed through observations.Source: §A.1 · p. 11
• The simulator applies accepted actions to update state; rejected actions leave the state unchanged.The simulator returns an execution-error observation when an action is rejected.Source: §A.1 · pp. 11–12

A.2 Planning and Reasoning Methodologies

• The framework separates direct action generation from formalization for a solver, and conservative reasoning from commitment based on hypothesized facts.Conservative reasoning uses verified observations; aggressive reasoning can fill gaps with assumed facts and commits to the full problem.Source: §A.2 · pp. 13–14

B Additional Computation Information and Experimental Setup

• Coin Collector allows k + 1 attempts for k hidden connections, ALFWorld allows 20, and formalizers can make up to 3 PDDL revision attempts.The limits represent worst-case logical attempts under partial observability, not API-call budgets.Source: B Additional Computation Information… · p. 14
• API calls count every LLM call, while an attempt ends in success or a rejected action; input cost grows with API calls because each call resends the task and history.Conservative plan extensions and PDDL revisions count as calls; hidden reasoning tokens are excluded from output-token counts.Source: B Additional Computation Information… · p. 14
• The experiments used DeepSeek-Flash through the default OpenAI API, Gemini-3-Flash through Agent Platform, and Qwen3.8-27B through vLLM.Gemini-3-Flash and Qwen3.8-27B used temperature 0.4; DeepSeek-Flash used default parameters.Source: B Additional Computation Information… · pp. 14–15

C.1 Multiple Runs

• Across three DeepSeek-Flash Coin Collector runs, metric variation was small relative to method differences and qualitative comparisons stayed consistent; the other models were run once.Figure 9 presents the repeated-run results; conservative methods lacked statistical priors, so their results are repeated.Source: §C.1 · p. 15, Figure 9 · p. 15
• Table 1 reports Gemini-3-Flash ALFWorld results for 10 problems per method using accuracy, failed attempts, API calls, and output tokens.Source: Table 1 · p. 15

C.2 ALFWorld Gemini Experiments

• On the 10-problem ALFWorld subset, Gemini-3-Flash performance was similar to DeepSeek-Flash and Qwen3.8-27B, except Conservative Planner used fewer API calls than Aggressive Planner, possibly because of the small sample.Source: §C.2 · p. 15

C.3 Non-LLM Coin Collector Baselines

• The non-LLM baselines include an oracle BFS with full map access, frontier BFS using observed information, aggressive BFS using deterministic guesses, and a stochastic policy based on dataset connection frequencies.The oracle is an upper bound rather than a directly comparable partially observable method.Source: §C.3 · pp. 15–16
• Except for Oracle BFS, non-LLM baselines required more attempts than every LLM method, while aggressive methods were more efficient or comparable to all non-LLM baselines.Table 2 reports attempts, steps, accuracy, and first-attempt success for the non-LLM baselines.Source: §C.3 · p. 16, Table 2 · p. 16

D Coin Collector Data

• The Coin Collector materials include a custom simulation, domain and action-head descriptions, statistical priors, and fully and partially observable problem descriptions.Source: D Coin Collector Data · pp. 16–17

E ALFWorld Household Data

• The ALFWorld household dataset selects rooms and receptacle-object pairs from 100 source worlds, then assigns each problem a distinct labeled five-room tree.Across the dataset, the worlds, goals, and labeled topologies are unique.Source: E ALFWorld Household Data · p. 17
• ALFWorld descriptions omit room connections and object locations; observations reveal current-room receptacles and adjacent-room names, while container contents may require inspection.Source: E ALFWorld Household Data · p. 17

F Prompts

• The prompts cover aggressive and conservative planners and formalizers, plus free and fixed hybrid variants; statistical priors are presented as “Common Sense Knowledge.”Source: F Prompts · p. 17

G.1 Direction-Based Move Execution

• In Coin Collector, aggressive moves mismatched the named destination in 809 of 3,953 rounds (20%); 805 of those rounds later ended in rejection, so direction-based execution delayed detection by a step.Only 4 of 1,141 successful aggressive rounds contained a mismatch; conservative methods had mismatches in 100 of 2,693 segments and 2 of 575 successful segments.Source: §G.1 · p. 17

G.2 Formalizer Error Analysis

• Formalizer failures are classified as PDDL, location, known-connection, or assumed-connection errors; assumed failures rely on connections not yet observed.The breakdown accounts for failed attempts in Tables 5–7.Source: §G.2 · pp. 17–18
• Aggressive formalizer failures were mostly wrong assumptions, reduced by priors by 42–46%; conservative DeepSeek-Flash failures were mostly stale locations, while Qwen3.8-27B failures were mostly PDDL errors.Stale locations accounted for 96–97% of DeepSeek-Flash conservative failures; Qwen3.8-27B needed about one revision call per problem, versus at most 0.12 for DeepSeek-Flash.Source: §G.2 · p. 18
• For Gemini-3-Flash, priors increased contradictions with known facts from 0.47 to 0.79 per problem, including cases where the model reasserted a high-probability connection already observed absent.Source: §G.2 · p. 18

G.3 Results by Path Length

• API calls and aggressive-method failed attempts rise with path length, while conservative planners’ failures stay near zero; both sides of the safety-efficiency tradeoff widen on longer paths.The problems have shortest paths of 2 moves in 42 cases, 3 in 30, 4 in 13, and 5–6 in 15, with mean 3.09.Source: §G.3 · p. 18
• For planners, priors reduce API calls by 19–41% at every path length.Source: §G.3 · p. 18
• For formalizers, API-call savings from priors are largest on short paths and nearly vanish at length 4, where few problems are available.Source: §G.3 · p. 18

• Tables 5–10 display the full results for all experiments performed in the paper’s main body.Source: H Full Results · p. 18
• In one example, the Conservative Planner solves the problem in one attempt, while the Aggressive Planner uses fewer API calls.Source: Figure 11 · p. 19
• With statistical priors under partial observability, the Aggressive Formalizer uses the priors and deductive reasoning to produce the correct plan.Source: Figure 12 · p. 19

H Full Results

• Conservative planner and formalizer prompts forbid assumptions and request a tangible subgoal when the supplied information is insufficient.The replanning prompts also direct the model to continue from the failed step rather than restart from the initial state.Source: H Full Results · pp. 26–27
• Hybrid planner prompts allow either a tangible subgoal or an inference about missing information, while hybrid PDDL prompts instruct the model not to assume information.Hybrid replanning begins at the failed step, and hybrid PDDL modification uses new observations.Source: H Full Results · pp. 28–29
• Hybrid planner replanning prompts ask for a repair from the failed step, allowing missing information to be inferred or a tangible subgoal to be proposed.They provide the prior plan and trajectory, and specify that replanning should not restart from the initial state.Source: H Full Results · pp. 30–31
• Conservative planner prompts prohibit assumptions and call for a tangible subgoal when the problem cannot be solved from the supplied information.The replanning version starts at the failed step rather than the initial state.Source: H Full Results · p. 32
• Hybrid formalizer prompts request PDDL revision or regeneration after errors, using the problem description, action set, prior files, plan, and trajectory.They allow proposing a tangible subgoal or inferring missing information when needed.Source: H Full Results · p. 33

Plan and observation updates

• For incorrect plans or new observations, prompts request updated PDDL, with some instructing the model to resume from the failed step rather than restart.Other variants revise the files using the plan trajectory and new observations.Source: H Full Results · pp. 36, 38

Initial formalization prompts

• Initial formalization prompts provide domain and problem descriptions, available actions, observations, and sometimes common-sense statistics, asking the model to infer missing information.The prompts request minimal PDDL files.Source: H Full Results · p. 37

Error revision prompts

• Revision prompts ask the formalizer to correct PDDL errors using the provided descriptions, prior files, and error message.Some variants also include common-sense knowledge.Source: H Full Results · pp. 37–38

H Full Results

• Tables 5–9 cover Coin Collector results for three LLMs across runs 1–3, reporting accuracy, attempts, failed attempts, API calls, and output tokens on 100 problems per row.The captions state that conservative methods are not given statistics.Source: Table 5 · p. 42, Table 6 · p. 42, Table 7 · p. 43
• Table 10 covers ALFWorld results for DeepSeek-Flash and Qwen3.8-27B without statistics, with 100 problems per method and the same five metrics.Source: Table 10 · p. 44