Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

1 INTRODUCTION

• The Plan Declaration–Execution Gap separates deviations during execution from selecting a plan poorly matched to the task, and final-success evaluation alone may not reveal either process.Existing evaluations largely focus on final outcomes, which do not show whether an agent followed a deliberate plan or reached the goal through another trajectory.Source: §1 · pp. 2–3
• The study asks whether agents preserve declared planning structure, whether pattern effectiveness varies across environments and models, whether matched executors improve success, and whether LLMs select suitable modes.The questions distinguish execution faithfulness from mode effectiveness and mode selection.Source: §1 · p. 3
• Planning-as-Routing has the LLM declare a mode and a deterministic router dispatch the task to its corresponding predefined, sequential, hierarchical, or search executor.The evaluation compares this framework with no-planning Flat ReAct and Plan+ReAct, where a generic ReAct executor receives a plan without structurally enforcing it.Source: §1 · p. 3
• The paper evaluates multiple LLM families across four benchmarks spanning web browsing, software engineering, and embodied tasks, and measures plan quality, adherence, and structural faithfulness beyond task success.Its contributions include diagnosing plan-execution divergence and reporting that current mode declarations trail the best fixed pattern on every benchmark–model pair; few-shot demonstrations improve success by +0.004 to +0.16 across benchmarks.Source: §1 · p. 3

LLMs as Planning Agents

• Prior planning approaches include ReAct, search, symbolic planning, planner–executor systems, multi-agent workflows, and adaptive replanning, but their planning mechanisms are typically fixed by the framework.This work instead treats planning mode as a per-task choice and asks whether the declared mode survives execution.Source: §2 · p. 3

Agent Evaluation and Trajectory Diagnosis

• Agent benchmarks mainly measure final outcomes, while recent process-level evaluations examine progress, plan compliance, trajectory diagnosis, and failure taxonomies.The paper complements this work by modeling the planner–executor handoff to distinguish mode-selection failures from failures to preserve the declared mode.Source: §2 · p. 4

3.1 PLANNING PATTERNS.

• The four planning patterns are predefined plans without replanning, sequential stepwise plans updated from observations, hierarchical subgoal decomposition, and search over independently executed candidates.A rubric-based judge selects the most promising search trajectory.Source: §3.1 · p. 4

3.2 PROBLEM FORMULATION.

• Flat ReAct acts without an explicit plan, Plan+ReAct gives a declared plan to a generic executor, and Planning-as-Routing dispatches the declared mode to a matching executor.The deterministic router makes the selected pattern determine execution control flow; the study also evaluates mode declaration with and without few-shot examples.Source: §3.2 · p. 4

3.3 PLANNING PROCESS METRICS

• Plan quality measures task appropriateness, plan adherence measures whether actions complete declared steps, and plan-order faithfulness checks whether steps or subgoals occur in declared order.Plan-order faithfulness applies to predefined, sequential, and hierarchical patterns, but not search, where candidates are alternatives rather than an ordered sequence.Source: §3.3 · p. 5

3.4 PATTERN-CEILING ANALYSIS

• Forced dispatch runs every planning mode on every task to estimate each fixed mode’s success, the best fixed-mode performance, a per-task oracle ceiling, and routing headroom.Comparing declaration-policy success with the same forced-dispatch matrix separates whether a selected mode is effective from whether execution preserves it.Source: §3.4 · p. 5

Environments

• Experiments span Mind2Web, WebArena, SWE-bench Verified, and ALFWorld, covering web navigation, software engineering, and embodied tasks.Task categories are also used to examine whether planning-pattern preferences vary with task type.Source: §4 · p. 5

Language Models

• The study evaluates Qwen3.6-35B-A3B, DeepSeek-V4-Flash, and Gemma-4-26B using the same task inputs, tool interfaces, action spaces, and execution budgets across conditions.The same backbone is used for declaration and execution unless otherwise specified.Source: §4 · p. 5

Evaluation Metrics

• Table 1 reports the percentage of generic Plan+ReAct trajectories preserving declared structure overall and by plan length, while omitting pattern-specific executors that enforce structure by design.Values are means ± SD across seeds 7, 13, and 42; sparse bins warrant caution.Source: Table 1 · p. 6

Repeated runs and Evaluation Metrics

• Runs use three random seeds, and evaluation reports benchmark-specific task-success metrics alongside plan quality, adherence, structural faithfulness, and execution-cost measures.The reported task metrics include TSR, Mind2Web SSR, and SWE-bench Verified PSR; cost measures include interactions, calls, tokens, and completed trajectories.Source: §4 · p. 6

Development and tuning protocol

• The planning prompts and executors are fixed before final evaluation, benchmark tasks are used only for inference, and few-shot demonstrations are excluded from scoring.Structural fidelity is checked by a rule-based verifier validated against two human annotators; quality and adherence use an LLM judge with a 0–3 rubric.Source: §4 · p. 6

[RQ1] Pattern-specific execution preserves declared planning structure

• 22–45% of Plan+ReAct trajectories preserve structure across ALFWorld, Mind2Web, and SWE-bench; fidelity falls with plan length and is typically below 25% for hierarchical plans.On ALFWorld, maintenance falls from 36.5–49.6% for 4–5-step plans to 4.8–21.4% for 6–7-step plans and approaches zero for longer plans.Source: [RQ1]: PATTERN-SPECIFIC EXECUTION… · pp. 6–7, Figure 2 · p. 7

Plan adherence and completion

• Structural fidelity and plan completion measure different properties: completion also depends on plan size and the fixed interaction budget.On ALFWorld, larger Hierarchical plans contain 10–13 executable units and achieve about 79–85% adherence; short Sequential plans are completed almost entirely.Source: [RQ1]: PATTERN-SPECIFIC EXECUTION… · pp. 7–8, Table 2 · p. 8

[RQ2] Planning-pattern effectiveness varies across environments and models

• The strongest planning pattern varies by benchmark–model pair; the forced-pattern oracle’s apparent headroom largely shrinks under retry-matched comparisons.After three retries of the strongest fixed pattern, the residual oracle gap is −0.025 to +0.027 across ALFWorld, Mind2Web, and SWE-bench; the oracle is an empirical ceiling, not direct evidence of task-specific complementarity.Source: [RQ2]: PLANNING-PATTERN EFFECTIVENESS… · p. 8, Table 3 · p. 8

[RQ3] Pattern-specific execution substantially improves over plan-as-context

• 0.480 with Plan+ReAct rises to 0.918 with the strongest fixed pattern on ALFWorld for DeepSeek-V4; on SWE-bench Verified, success rises from 0.360 to 0.442.Across benchmarks, generic Plan+ReAct yields limited gains over Flat ReAct, while pattern-specific execution achieves substantially higher success.Source: Table 5 · p. 9, [RQ3]: PATTERN-SPECIFIC EXECUTION… · p. 10

Planning quality and task success

• Plan quality and task success are only weakly correlated (|r| ≤ 0.181), and the highest-quality plan does not consistently belong to the highest-performing pattern.For DeepSeek-V4 on ALFWorld, Sequential has the highest judged plan quality (2.84±0.03), while Search has higher task success (0.91±0.03 versus 0.64±0.02).Source: [RQ2]: PLANNING-PATTERN EFFECTIVENESS… · p. 9, Table 4 · p. 9

[RQ1] Pattern-specific execution preserves declared planning structure

• Pattern-specific execution’s advantage is largest on longer tasks, where Plan+ReAct and Flat ReAct degrade more rapidly.The separation is modest on short tasks and widens as more interaction steps are required; the pattern is similar on WebArena, while pattern-specific executors remain competitive with or above Plan+ReAct across most Mind2Web length bins.Source: Figure 3 · p. 10, [RQ3]: PATTERN-SPECIFIC EXECUTION… · p. 11

Search rollout-matched control

• Against 3× independent Flat ReAct runs with the same trajectory judge, Search remains ahead by 0.117–0.381 on ALFWorld and 0.153–0.233 on WebArena.Differences are smaller and model-dependent on Mind2Web.Source: [RQ3]: PATTERN-SPECIFIC EXECUTION… · p. 11

[RQ4] Planning-mode declarations adapt only weakly to individual tasks

• Declarations show no reliable task-specific advantage over a task-blind policy, while few-shot demonstrations improve top-1 selection unevenly across benchmarks.Ranked fallback raises task success as more declared modes are attempted, but adds execution attempts.Source: Table 6 · p. 11, [RQ4]: PLANNING-MODE DECLARATIONS… · p. 12

6 DISCUSSION AND CONCLUSION

• Generic Plan+ReAct often fails to preserve declared structure, pattern effectiveness varies across environments and models, and matching patterns to executors improves success, while mode selection remains challenging.The discussion also reports that retry-matched controls suggest the raw forced-pattern oracle largely reflects extra execution attempts rather than task-specific pattern complementarity.Source: §6 · p. 12
• Reliable agent planning requires selecting an appropriate mode and executing it through a compatible control structure.Planning-as-Routing separates the two capabilities to distinguish mode-selection failures from execution failures.Source: §6 · p. 12
• The study covers four planning patterns and task-level routing under fixed execution budgets; future work could add composed or dynamically switchable strategies.The authors also propose jointly evaluating structural fidelity with action correctness, execution cost, and recovery behavior.Source: §6 · p. 12

A ADDITIONAL RELATED WORK

• Earlier planning approaches generally fix the planning mechanism in the agent framework, whereas this work separates declared mode from execution architecture.The comparison frames the work as testing whether execution preserves the declared mode.Source: A ADDITIONAL RELATED WORK · p. 18
• The paper introduces handoff-aware failure attribution across planner, router, executor, and verifier to distinguish inappropriate mode choice from unfaithful execution.This contrasts with trajectory analyses that diagnose failures after execution without explicitly modeling the handoff.Source: A ADDITIONAL RELATED WORK · p. 18

B PLANNING PATTERNS

• The four patterns are predefined fixed plans, sequential replanning, hierarchical subgoal coordination, and search over independently executed candidates judged by a rubric.A rubric-based judge selects the most promising resulting trajectory for Search.Source: B PLANNING PATTERNS · p. 18

C DATASET AND MODEL DETAILS

• The evaluation spans four agent benchmarks covering web interaction, software engineering, and embodied task execution.Table 7 summarizes evaluated task counts, task characteristics, and evaluation metrics.Source: C DATASET AND MODEL DETAILS · p. 18
• The study evaluates Qwen3.6-35B-A3B, DeepSeek-V4-Flash, and Gemma-4-26B-A4B-it, using the same backbone for declaration and execution within each run.Model identity is held fixed across execution-condition comparisons unless stated otherwise.Source: C DATASET AND MODEL DETAILS · p. 18
• The illustrated executors include fixed ordered plans, stepwise execution with replanning, and orchestrated subgoal decomposition with workers.The figure labels these predefined, sequential, and hierarchical planning, respectively.Source: Figure 4 · p. 19

D INFERENCE AND SAMPLING CONFIGURATION

• Decoding, thinking configuration, tool-calling policy, and generation budget are held fixed across Flat ReAct, Plan+ReAct, and Planning-as-Routing for each backbone.This controls inference-level differences in comparisons of planning and execution mechanisms.Source: D INFERENCE AND SAMPLING CONFIGURATION · p. 19
• Search generates multiple candidate plans for independent execution and uses a task-specific rubric judge, capped at four candidates per task.Across Search runs, the realized candidate count was 3.35 ± 0.49 over 15,598 episodes; the judge lacks ground-truth success labels and hidden evaluation scores.Source: Figure 5 · p. 20, D INFERENCE AND SAMPLING CONFIGURATION · p. 22
• The benchmark and model tables identify the evaluation sets and backbones, while Table 9 describes task-category composition and generalization or difficulty groupings.The supplied captions establish table scope but do not provide the underlying cell values.Source: Table 7 · p. 20, Table 8 · p. 20, Table 9 · p. 21
• The common generation profile allows 32,768 new tokens per turn and requires a 16,384-token minimum per turn for thinking models.The effective budget is described as model and profile budgets subject to the minimum thinking floor.Source: D INFERENCE AND SAMPLING CONFIGURATION · p. 20, Table 11 · p. 21

E.1 PLAN-STRUCTURE VERIFIER

• The deterministic verifier filters non-actionable plan text and matches executable units to actions in declared order, allowing intervening environment actions.Structural preservation is distinct from action correctness; routed executors are not scored with the matcher because dispatch records establish control flow.Source: §E.1 · p. 22

E.2 REPRESENTATIVE PLAN+REACT TRACES

• The trace examples use ordered matching of scorable plan units as the criterion for preserved structure, despite extra actions between units.The examples compare declared plans with action sequences produced by generic Plan+ReAct.Source: §E.2 · pp. 22–23
• The supplied traces illustrate sequential, predefined, hierarchical, and search plans through declared units and their corresponding executed actions.The examples respectively cover pillow placement, mug cleaning, peppershaker placement, and competing tomato-location paths.Source: Sequential · p. 23, Predefined · p. 23, Hierarchical · p. 23

E.3 HUMAN VALIDATION OF THE STRUCTURE VERIFIER

• κ=0.31 against A1 and κ=0.34 against A2, close to human–human κ=0.35 on 100 sampled trajectories.The two human annotators achieved 71% raw agreement, indicating that judging free-form trajectory structure can itself be ambiguous.Source: §E.3 · p. 24

F PLAN QUALITY JUDGE

• The plan-quality judge uses a common evaluator across DeepSeek and Qwen ALFWorld sweeps and avoids self-judging Gemma runs.Scores from different judge models are not pooled; plan-quality comparisons use a common judging configuration.Source: F PLAN QUALITY JUDGE · pp. 24–25

Plan Quality: goal + initial plan only; execution withheld

• Plan quality is judged from the goal, available tools, and initial written plan, without using execution information.The prompt asks evaluators to assess the plan as a strategy at the time it was written, rather than from its eventual success.Source: Plan Quality: goal + initial plan… · p. 25
• The plan-quality criteria cover goal coverage, tool selection and feasibility, and whether steps are actionable and logically ordered.The evaluator assigns a single score from 0 to 3 and assesses the plan's efficiency and use of available information.Source: Plan Quality: goal + initial plan… · p. 25

Plan Adherence: declared plan + execution trajectory

• Plan adherence evaluates completion of executable plan units using trajectory evidence, not intrinsic plan quality or final task success.Extra actions do not count against adherence unless they replace or prevent completion; valid replanning changes the active plan being assessed.Source: Plan Adherence: declared plan +… · pp. 25–26
• The adherence prompt scores execution from 0 to 3 based on how completely important executable units are followed and supported by trajectory evidence.The procedure first identifies the plan, decomposes it into executable units, and treats hierarchical leaf-level units as the evaluation units.Source: Plan Adherence: declared plan +… · pp. 25–26

• Hierarchical maintenance on ALFWorld was 11.4%, 4.6%, and 0.8% for DeepSeek-V4, Qwen3.6-35B, and Gemma-4-26B; Mind2Web rates were 14.5%, 17.3%, and 22.6%.Source: G STRUCTURE PRESERVATION BY DECLARED… · p. 26
• On SWE-bench, Hierarchical maintenance was 17.3% for DeepSeek-V4 and 14.5% for Qwen3.6-35B.Source: G STRUCTURE PRESERVATION BY DECLARED… · p. 26
• Search is also difficult for generic ReAct to preserve where enough declarations are observed, but some Search cells are sparse and its candidate structure differs from an ordered plan.Together with plan-length analysis, these results indicate that generic ReAct poorly maintains richer multi-level or multi-candidate structures.Source: G STRUCTURE PRESERVATION BY DECLARED… · p. 26

• Plan adherence measures completion of still-needed plan units within the available interaction budget, and depends on both budget and plan size.Source: H ADDITIONAL PLAN ADHERENCE RESULTS · p. 26
• Mind2Web browser tasks often require prerequisite interactions, making skipped necessary units consequential while distinguishing needed steps from redundant or obsolete ones.Source: H ADDITIONAL PLAN ADHERENCE RESULTS · p. 27
• The Mind2Web table reports planned and completed units per task, adherence, and the fraction of trajectories completing all needed units; Search values are not directly comparable to ordered patterns.Values are means ± standard deviations across seeds 7, 13, and 42.Source: Table 15 · p. 27
• On SWE-bench Verified, Hierarchical plans declare 13.7 and 9.4 units per task, versus 3.3–5.5 for other patterns, yet complete the smallest fraction.Both models ran at seed 13 only; values have no reported spread.Source: Table 16 · p. 28

I PATTERN-CEILING ANALYSIS

• Forced dispatch runs all four patterns on every task, yielding a task–pattern success matrix from which fixed-pattern performance and the per-task oracle ceiling are derived.This bypasses the declaration module to distinguish whether a pattern can solve a task from whether it is selected.Source: I PATTERN-CEILING ANALYSIS · p. 28

I.2 REPEATED-FIXED-PATTERN CONTROL.

• The declaration null preserves each policy's overall mode frequencies but randomly reassigns declarations across tasks; the reported delta compares policy TSR with this task-blind baseline.The table reports a 95% permutation interval, one-sided and two-sided p-values, and Holm-corrected p-values.Source: I PATTERN-CEILING ANALYSIS · p. 28, Table 19 · p. 32

I PATTERN-CEILING ANALYSIS

• A trace verifier identifies dispatch failure when execution differs from declaration, while faithful dispatch of a wrong pattern indicates selection failure.Forced-pattern results are averaged across independently sampled executions by seed because execution is stochastic.Source: I PATTERN-CEILING ANALYSIS · p. 29

I.1 REPEATED-ROLLOUT REACT CONTROL FOR SEARCH.

• Search exceeded 3×Flat ReAct + Judge by 0.132–0.392 on ALFWorld and 0.128–0.191 on WebArena; repeated ReAct slightly outperformed Search on SWE-bench.On SWE-bench, repeated ReAct versus Search scored 0.410 versus 0.400 for DeepSeek-V4 and 0.304 versus 0.296 for Qwen3.6-35B; Mind2Web results were smaller and model-dependent.Source: Table 17 · p. 29, §I.1 · pp. 29, 31
• Search averages 3.35 ± 0.49 candidate plans per task, so the control runs three independent Flat ReAct trajectories and applies Search's rubric-based judge.The repeated-rollout comparison tests how much Search's advantage can be reproduced through generic ReAct retries and trajectory selection.Source: §I.1 · p. 29, Table 17 · p. 29
• Pass@3 hindsight selection over the same three Flat ReAct trajectories separates attainable candidate success from the judge-selected outcome and tests candidate quality against Search.Source: §I.1 · p. 31

I.2 REPEATED-FIXED-PATTERN CONTROL.

• The fixed-pattern retry control compares repeated executions of the strongest fixed pattern against the forced-pattern oracle; the Mind2Web control uses TSR and SSR.For Gemma-4-26B on Mind2Web, Predefined is the TSR-best fixed pattern, while Search has the highest single-run SSR at 0.388; reported SSR retries still use Predefined.Source: §I.2 · p. 31, Table 18 · p. 31

J COMPARISON WITH PRIOR WORK.

• Prior-system scores provide benchmark-level context, not controlled comparisons, because backbones, demonstrations, training, and inference procedures differ.The primary comparisons instead use controlled Flat ReAct, Plan+ReAct, and Planning-as-Routing conditions under the same models and execution protocol.Source: J COMPARISON WITH PRIOR WORK · p. 32

K ADDITIONAL ANALYSIS

• ∆(π) ranged from −0.025 to +0.009, with no benchmark–model pair showing a significant advantage over the task-blind policy.The null preserves overall mode frequencies while removing their association with individual tasks.Source: K ADDITIONAL ANALYSIS · p. 32
• 0.948 (DeepSeek-V4 TSR@3 on ALFWorld) exceeds TSR@1’s 0.721, while TSR@k success increases across the reported benchmarks as k rises.On ALFWorld, DeepSeek-V4 reaches 0.893 at TSR@2; on SWE-bench it rises from 0.415 to 0.541 across TSR@1 to TSR@3.Source: K ADDITIONAL ANALYSIS · pp. 32–33
• 0.978 (DeepSeek-V4 pass@3 on ALFWorld) exceeds TSR@3’s 0.948, showing that repeated execution of a strong fixed pattern can obtain much of fallback’s gain.On SWE-bench, TSR@3 and pass@3 are nearly identical for DeepSeek-V4: 0.541 versus 0.538.Source: K ADDITIONAL ANALYSIS · p. 33
• Enabling thinking changes planning-mode preferences but does not consistently improve top-1 declaration success across models and benchmarks.For DeepSeek-V4, TSR@1 decreases on all four benchmarks; Qwen3.6-35B improves only on ALFWorld among the reported comparisons.Source: K ADDITIONAL ANALYSIS · p. 33
• Few-shot task–pattern demonstrations improve top-1 performance in most configurations, with gains varying by benchmark and model.The largest reported increase is DeepSeek-V4 on ALFWorld with thinking enabled, from 0.656 to 0.812 (+0.156).Source: K ADDITIONAL ANALYSIS · p. 33
• On Mind2Web, better mode declaration alone may yield limited downstream gains because grounding and operation-selection errors can persist across the trajectory.Website and task diversity may also make a small set of demonstrations less informative in this environment.Source: K ADDITIONAL ANALYSIS · p. 33

L INFERENCE COST ANALYSIS

• Hierarchical and Search generally use more aggregate tokens than Flat ReAct, Plan+ReAct, Sequential, and Predefined execution, largely through more frequent LLM calls.Search often makes the most calls but uses fewer tokens per call than Hierarchical execution.Source: L INFERENCE COST ANALYSIS · p. 33
• On ALFWorld, Search uses fewer aggregate tokens than Hierarchical for DeepSeek-V4 and Qwen3.6-35B while achieving higher task success.Neither total token usage nor LLM-call count tracks task success monotonically.Source: L INFERENCE COST ANALYSIS · p. 33, Figure 7 · p. 34
• Greater aggregate token usage does not consistently correspond to higher task success across the four benchmarks.Figure 7 reports mean generated tokens per task, divided into thinking and answer/content tokens, with standard-deviation error bars.Source: L INFERENCE COST ANALYSIS · p. 33, Figure 7 · p. 34

M EXTENDED DISCUSSION

• Generic Plan+ReAct preserves declared structure in only a minority of trajectories, with fidelity falling for longer plans; pattern-specific executors enforce their structures.Structural faithfulness is distinct from plan adherence: preserving a plan’s organization does not ensure that all of it is completed within the interaction budget.Source: M EXTENDED DISCUSSION · pp. 35–36
• No planning pattern dominates across environments and models; plan quality weakly predicts success, and retries by the strongest fixed pattern explain much of the per-task oracle gap.Search, Hierarchical, and Predefined are strongest in different benchmark–model settings.Source: M EXTENDED DISCUSSION · p. 36
• Pattern-specific execution yields larger gains than supplying an explicit plan to generic ReAct, especially on longer tasks; Search’s advantage is not generally explained by extra attempts.The discussion contrasts modest gains from explicit plans in the same generic loop with substantially larger gains from the corresponding pattern-specific architecture.Source: M EXTENDED DISCUSSION · p. 36
• Current top-ranked declarations provide no reliable task-specific advantage, while few-shot demonstrations improve top-1 performance in all but one evaluated configuration.The demonstrations are built from forced-pattern executions, retaining tasks where the same single mode succeeds in both runs; additional declaration reasoning does not consistently improve top-1 performance.Source: M EXTENDED DISCUSSION · p. 36, Shared Planning-Mode Declaration… · p. 38
• The study’s pattern ceiling is empirical, and its oracle is not pure selection headroom because it combines executions from multiple patterns.The observed ceiling and relative pattern effectiveness may change with models, executors, tools, or execution budgets.Source: M EXTENDED DISCUSSION · pp. 36–37
• Because declaration occurs once per task, future work could explore dynamic routing that switches patterns as task state changes while preserving interpretable traces.One example is switching from hierarchical decomposition to sequential replanning after an unexpected observation.Source: M EXTENDED DISCUSSION · p. 37
• Structural verification does not establish action correctness or plan optimality, motivating evaluation that combines structural fidelity with action correctness, cost, recovery, and task success.Handoff-aware evaluation could also distinguish failures in planning, routing, delegation, communication, or execution in multi-agent systems.Source: M EXTENDED DISCUSSION · p. 37

Few-shot declaration prompt (SWE-bench)

• The SWE-bench prompt presents prior tasks and their best-mode plans as illustrations of Predefined, Sequential, Hierarchical, and Search—not as a quota or ranking rule.Examples show fixed-sequence, stepwise, nested-phase, and competing-route plans; the prompt says any mode may be right for the new task.Source: Few-shot declaration prompt (SWE-bench) · p. 51