PaperlayerView exact revision on arXiv

Source-linked AI summary

On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability

Kevin Wang, Junbo Li, Neel P. Bhatt, Yihan Xi, Qiang Liu, Ufuk Topcu, Zhangyang Wang

arXiv:2409.19924v4cs.AIcs.LGcs.RO

TL;DR

LLM planning remains underexplored, and prior scrutiny finds that models struggle to make successful plans without external tools. This pilot evaluates o1 across planning dimensions and tasks, finding stronger constraint-following across many tasks alongside weaker reliability as spatial complexity rises.

  • Problem

    Although LLMs perform well in language-related tasks, their effectiveness in planning remains underexplored, and prior scrutiny reports difficulty making successful plans without external tools.

    Source: §1 · p. 1

  • Method

    The study evaluates o1 across diverse planning tasks, classifying errors through feasibility, optimality, and generalizability rather than success rate alone.

    Source: §1 · p. 2

  • Results

    o1-preview follows constraints better than GPT-4 across most tasks, particularly in rule-based settings, but constraint-following and state management weaken as spatial complexity rises.

    Source: §5.1 · p. 12

  • Takeaways & Limitations

    The pilot offers foundational insight into o1’s planning strengths and limitations and supports finer-grained diagnostics of planning errors.

    Source: §1 · p. 2, §5 · p. 12

    The empirical evaluation uses a relatively small dataset; broader generalizability and robustness require larger, more diverse tests, including real-world settings with dynamic, less predictable elements.

    Source: §4.1 · pp. 9–10

Abstract

from arXiv · show

Recent advancements in Large Language Models (LLMs) have showcased their ability to perform complex reasoning tasks, but their effectiveness in planning remains underexplored. In this study, we evaluate the planning capabilities of OpenAI's o1 models across a variety of benchmark tasks, focusing on three key aspects: feasibility, optimality, and generalizability. Through empirical evaluations on constraint-heavy tasks (e.g., $\textit{Barman}$, $\textit{Tyreworld}$) and spatially complex environments (e.g., $\textit{Termes}$, $\textit{Floortile}$), we highlight o1-preview's strengths in self-evaluation and constraint-following, while also identifying bottlenecks in decision-making and memory management, particularly in tasks requiring robust spatial reasoning. Our results reveal that o1-preview outperforms GPT-4 in adhering to task constraints and managing state transitions in structured environments. However, the model often generates suboptimal solutions with redundant actions and struggles to generalize effectively in spatially complex tasks. This pilot study provides foundational insights into the planning limitations of LLMs, offering key directions for future research on improving memory management, decision-making, and generalization in LLM-based planning. Code available at https://github.com/VITA-Group/o1-planning.

1 Introduction

Planning by language agents remains underexplored, and prior scrutiny finds that LLMs struggle to make successful plans without external tools [Liu et al., 2023, Valmeekam et al., 2024a, 2023, 2024b]. This study examines o1 across a broader range of planning tasks and classifies errors across feasibility, optimality, and generalizability.

1 Introduction

  • Prior scrutiny reports that LLMs struggle to make successful plans without external tools such as a PDDL planner [Liu et al., 2023, Valmeekam et al., 2024a, 2023, 2024b].The cited work evaluates reasoning approaches including Chain-of-Thought and Tree-of-Thought.Source: §1 · p. 1
  • Figure 1 compares GPT-4, o1-mini, and o1-preview across the paper’s key planning perspectives.Source: Figure 1 · p. 1, §1 · p. 2
  • The authors ask where o1 still falls short and how its planning limitations can be identified systematically.This question follows reported improvements in o1’s planning capabilities.Source: §1 · p. 2
  • The study analyzes o1 across diverse planning tasks and classifies errors by feasibility, optimality, and generalizability rather than measuring success rates alone.It aims to provide finer-grained diagnostics of limitations across the planning pipeline.Source: §1 · p. 2

2 Planning Ability Evaluation: Three Perspectives

The evaluation separates planning into feasibility, optimality, and generalizability, each addressing a distinct but connected capability. Table 1 reports that o1-preview has the highest success rate across all domains.

2 Planning Ability Evaluation: Three Perspectives

  • The framework evaluates feasibility, optimality, and generalizability as distinct but interconnected dimensions of planning.Together, the dimensions cover execution, efficiency, and adaptation across contexts.Source: §2 · p. 2
  • Feasibility covers executable, rule-compliant steps, a coherent goal-reaching plan, and correct interpretation of initial and goal states.The corresponding failure types are Inability to Follow Problem Rules (IR), Inability to Generate a Feasible Plan (IP), and Misinterpretation of Goal State (MG).Source: §2 · p. 2
  • Optimality concerns efficiency beyond successful execution; Lack of Optimality (LO) marks feasible plans containing extraneous or inefficient actions.Such actions prevent a plan from being considered the best solution.Source: §2 · pp. 2–3
  • To test generalization, the study uses arbitrary action symbols; o1-preview degrades on generalized tasks, especially in complex, spatially dynamic environments.The test probes whether plans remain valid without natural-language cues.Source: §2 · p. 3
  • o1-preview has the highest success rate across all domains in Table 1.The table reports counts for IP, LO, MG, and IR, alongside success rate (SR).Source: Table 1 · p. 3

3 Experiments on Planning Benchmarks

Across six tasks and three models, o1 improved success on some benchmarks, but recurring planning errors remained. The experiments examine feasibility, optimality, and generalizability across distinct planning domains.

3 Experiments on Planning Benchmarks

  • Across six tasks and three models, o1 improved success on some benchmarks, but recurring planning errors remained across feasibility, optimality, and generalizability.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Barman

  • In Barman, agents repeatedly violated action rules, including hand-free and empty-shot requirements, and o1-preview also struggled to produce feasible plans.Source: §3.1 · p. 5, Figure 5 · p. 5

3.2 Blocksworld

  • 100% success for o1-preview, versus 60% for o1-mini and 40% for GPT-4, still included an unnecessary action in one Blocksworld plan.Source: §3.2 · p. 6, Figure 6 · p. 6

3.3 Grippers

  • 90% success and 70% optimality for o1-preview; o1-mini reached 80% on both, while GPT-4 scored 70% and 20%, respectively.Despite its advantage, o1-preview misread a goal that was already satisfied, prompting unnecessary movement.Source: §3.3 · pp. 6–7, Figure 7 · p. 7

3.4 Floortile

  • All models failed Floortile; only 30% of o1-preview's failures stemmed from rule violations, as self-evaluation helped it track and correct constraints.Source: §3.4 · p. 7, Figure 8 · p. 8

3.5 Termes

  • All three models failed Termes, commonly ignoring height constraints or placing blocks at their current position rather than neighboring positions.Source: §3.5 · p. 8, Figure 9 · p. 9

3.6 Tyreworld

  • 100% to 20%: o1-preview's Tyreworld success fell when actions and tools became random symbols, despite unchanged underlying constraints and logic.It generated correct plans for all test problems; GPT-4 and o1-mini completed only the simplest cases.Source: §3.6 · p. 9

4 Discussion

o1-preview follows constraints and manages states better than GPT-4 in structured tasks, but spatial complexity weakens performance and plans often remain inefficient. Generalization is strongest in structured, low-dimensional tasks, while the study’s small dataset limits broader conclusions.

4.1 Empirical Limitations

  • The relatively small evaluation dataset makes broader conclusions about o1’s generalizability and robustness dependent on larger, more diverse testing.The authors also call for testing in real-world settings with dynamic and less predictable elements.Source: §4.1 · pp. 9–10

4.2 Model Performance vs. Problem Complexity

  • Performance correlated strongly with problem complexity: spatial relationships and state transitions were more problematic than action-space size.Floortile and Termes pose higher spatial and rule-based demands; Grippers illustrates that tasks with limited actions can remain manageable.Source: §4.2 · pp. 10–11, Figure 12 · p. 11

4.3 Constraint Following and State Management

  • o1-preview’s self-evaluation improved constraint following and state management in Blocksworld and Tyreworld, but these abilities weakened in spatially complex Termes.It outperformed GPT-4 and avoided rule violations that affected GPT-4 and o1-mini in the cited structured tasks.Source: §4.3 · p. 11

4.4 Optimality and Redundancy in Planning

  • o1 models often produced feasible but suboptimal plans, adding redundant actions that reduced efficiency in tasks such as Blocksworld and Floortile.The models also sometimes assumed rules that were not present, as in o1-preview’s Grippers room-movement assumption.Source: §4.4 · p. 11

4.5 Generalization and Adaptability

  • o1-preview generalized across tasks with consistent rule structures, as in Grippers, but remained limited on dynamic, high-dimensional, and abstract problems.Its adaptation to randomized symbols was imperfect, including a Tyreworld example where a nut was not loosened before being undone.Source: Figure 11 · p. 10, §4.5 · p. 11

5 Conclusion

This pilot evaluation finds strengths in constraint following and state management in structured tasks, alongside weaknesses in optimality and abstract-task generalization. It identifies decision-making, memory, and handling of complex environments as areas for further work.

5 Conclusion

  • The pilot evaluation identifies promising advances and continuing challenges across feasibility, optimality, and generalizability in OpenAI’s o1 models.The authors conclude that plan optimization, abstract-task generalization, and state complexity remain significant challenges.Source: §5 · p. 12, §5.2 · p. 13

5.1 Summary of Findings

  • o1-preview showed stronger constraint understanding in well-defined tasks, but its constraint following, state management, and generalization weakened in more spatially complex or abstract settings.The summary also identifies reliable multi-state memory as an advantage that becomes less dependable as complexity increases.Source: §5.1 · p. 12

5.2 Opportunities for Improvement

  • Cost-based reasoning, expert demonstrations, and continuous human feedback are proposed to reduce redundant actions and improve decision-making and adaptation.Interactive feedback could help models adjust to new situations or tasks that deviate from training data.Source: §5.2 · pp. 12–13
  • Retrieval-augmented generation could provide real-time external memory updates, but its effectiveness depends on accurate and efficient retrieval.The authors particularly note tasks relying on large knowledge bases encoded in natural-language text.Source: §5.2 · p. 12
  • Future work should improve generalization to abstract, high-dimensional, and spatially dynamic environments through better memory management and abstraction.The paper also identifies less direct links between actions, outcomes, and natural-language representations as a challenge.Source: §5.2 · p. 12, §5.1 · p. 12
  • Testing in dynamic, unpredictable settings would provide evidence about robustness and adaptability when rules or constraints change during execution.Source: §5.2 · p. 13
  • More robust self-evaluation, multi-stage validation, or symbolic verification is proposed to help catch constraint-related errors before decisions are finalized.Source: §5.2 · p. 13
  • Multimodal inputs such as visual or sensor data and decentralized multi-agent planning are proposed for complex spatial tasks and coordinated goals.The authors cite robotics and logistics as settings where multi-agent coordination is relevant.Source: §5.2 · p. 13

Related papers

Loading 2409.19924v4…