Source-linked AI summary
LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, Anil Murthy
TL;DR
The paper addresses confusion over whether LLMs can plan or self-verify and whether their role should be limited to translation. It argues for LLM-Modulo Frameworks that use LLMs as approximate knowledge sources and candidate generators alongside external sound verifiers, concluding that LLMs can support planning without independently planning or verifying.
Problem
The paper addresses conflicting claims about LLMs’ planning and self-verification abilities and about whether they contribute anything beyond translating problems for symbolic solvers.
Method
The paper reviews literature on LLM planning and verification, then proposes an LLM-Modulo Generate-Test-Critique framework combining LLMs with external critics, verifiers, and humans.
Results
The paper concludes that LLMs cannot plan or self-verify by themselves but can constructively generate candidate plans and provide approximate knowledge when externally checked.
Takeaways & Limitations
LLMs should be leveraged for constructive planning roles while external sound model-based verifiers provide formal correctness guarantees where possible.
Abstract
from arXiv · showhide
There is considerable confusion about the role of Large Language Models (LLMs) in planning and reasoning tasks. On one side are over-optimistic claims that LLMs can indeed do these tasks with just the right prompting or self-verification strategies. On the other side are perhaps over-pessimistic claims that all that LLMs are good for in planning/reasoning tasks are as mere translators of the problem specification from one syntactic format to another, and ship the problem off to external symbolic solvers. In this position paper, we take the view that both these extremes are misguided. We argue that auto-regressive LLMs cannot, by themselves, do planning or self-verification (which is after all a form of reasoning), and shed some light on the reasons for misunderstandings in the literature. We will also argue that LLMs should be viewed as universal approximate knowledge sources that have much more meaningful roles to play in planning/reasoning tasks beyond simple front-end/back-end format translators. We present a vision of {\bf LLM-Modulo Frameworks} that combine the strengths of LLMs with external model-based verifiers in a tighter bi-directional interaction regime. We will show how the models driving the external verifiers themselves can be acquired with the help of LLMs. We will also argue that rather than simply pipelining LLMs and symbolic components, this LLM-Modulo Framework provides a better neuro-symbolic approach that offers tighter integration between LLMs and symbolic components, and allows extending the scope of model-based planning/reasoning regimes towards more flexible knowledge, problem and preference specifications.
1. Introduction
The paper argues that LLMs cannot independently plan or verify, but can support planning as approximate knowledge sources and candidate generators within externally checked frameworks. It addresses both overestimating and underestimating LLMs’ roles in planning and reasoning.
- The paper’s position: The paper rejects both optimism about LLM reasoning competence and pessimism that limits LLMs to translating specifications for symbolic solvers.It instead treats LLMs as useful approximate knowledge sources without guarantees.
- View of LLMs: LLMs are characterized as giant external, non-veridical memories that can provide useful cognitive support without possessing System 2 competence themselves.Their broad training enables approximate knowledge and connections across disparate fields, but does not justify attributing capabilities they lack.
- The paper’s position: LLMs cannot independently perform planning or plan verification, despite continued claims that prompting or self-critique can elicit these abilities.The paper attributes this position to the limits of autoregressive LLM operation and reviews literature questioning their planning and verification capabilities.
- LLM-Modulo Framework: The proposed LLM-Modulo Framework combines LLMs with external critics, verifiers, and humans in a tighter planning interaction.LLMs can generate candidate plans, translate them for critics, help complete specifications, and assist experts in acquiring domain models.
- LLM-Modulo Framework: External critics vet LLM-generated ideas, allowing formal correctness guarantees for plans where such guarantees are possible.The framework therefore assigns planning and verification responsibility to external sound components rather than to the LLM alone.
2. Planning-centered Limitations of LLMs
The paper argues that LLMs do not reliably generate executable plans or verify them autonomously, including when prompted iteratively, because their apparent planning competence is not robust.
- Autonomous plan generation: Only about 12% of plans generated by GPT-4 are executable and goal-reaching without errors in autonomous mode.
- Autonomous plan generation: Fine-tuning and model choice have little effect on poor planning performance, while obfuscating action and object names degrades it further.
- Autonomous plan generation: Chain-of-thought and ReAct-style prompting are largely ineffective at improving LLM planning performance.
- Verification and self-critique: LLMs are no better at verifying graph colorings than solving them, so self-critique can replace correct solutions with incorrect ones.
- Analyzing contrary claims: Claims of planning and self-verification often rely on simplified tasks, ignored subgoal interactions, or external simulators and critics rather than autonomous LLM reasoning.
- Analyzing contrary claims: The paper instead positions LLMs as approximate knowledge sources whose extracted domain and preference models require human or specialized-critic verification before sound model-based planning.
3. LLM-Modulo Framework for Robust Planning
The LLM-Modulo Framework combines LLM-generated candidates with external critics and verifiers in a generate-test-critique loop. It uses LLMs for candidate generation, reformulation, approximate modeling, and specification support while reserving soundness judgments for external components.
- Framework architecture: LLM-Modulo uses a generate-test-critique loop in which an LLM proposes candidate plans and critics evaluate them.The system can return a candidate once all hard critics sign off; completeness depends on generating all potentially relevant candidates.
- Framework architecture: External critics and verifiers provide soundness guarantees, while LLM-generated candidates remain incomplete if relevant possibilities are never generated.The framework’s soundness comes from external sound critics, whereas completeness depends on the LLM’s candidate-generation ability.
- Design choices: The framework deliberately combines LLMs with composable external critics rather than using them only as frontends for external solvers.Constructive critics may themselves use solvers or partial planners to suggest specific plan modifications.
- LLM roles: LLMs can generate approximate plan ideas, domain models, problem reductions, specification refinements, and translations between specialized syntactic representations.Human experts can curate LLM-assisted domain models, while reformulators translate candidate plans for individual critics.
- Critics and feedback: Critics assess both hard constraints, such as causal, timeline, and resource correctness, and soft constraints, such as style, explicability, and preferences.Hard critics may use VAL or simulators; soft critics may be LLM-based but do not provide soundness guarantees.
- Critics and feedback: A meta-controller pools and processes heterogeneous critiques before sending consistent iterative feedback to the LLM.This controller is particularly useful when hard and soft critics provide different kinds of feedback.
- Scope and safety: Unlike narrow guaranteed planners, LLMs can suggest plausible heuristics across more flexible scenarios, provided plans are vetted rather than sent directly to execution.The architecture imposes fewer restrictions on knowledge, problem, and preference specifications than traditional formal planning architectures.
4. Two Case Studies of LLM-Modulo
The paper applies LLM-Modulo to classical planning and flexible natural-language travel planning. Reported results show improvement with verifier-guided back prompting, while the travel benchmark exposes the weakness of standard prompting strategies.
- Classical planning: 82% Blocks World performance was reached within 15 back-prompting rounds when VAL served as the external verifier and critic.This result comes from the paper’s reported classical-planning case study.
- Travel planning: 0.7% was the best reported GPT-3.5-Turbo performance on the travel benchmark using existing prompt-engineering strategies.The benchmark presents rich travel constraints in flexible natural-language form and includes strategies such as Chain of Thought and ReAct.
5. Related Work
Related work illustrates LLM-Modulo’s Generate-Test-Critique pattern across symbolic evaluation, reinforcement learning with simulators, multimodal criticism, and agentic systems. These examples position external evaluation as essential while distinguishing action invocation from planning.
- Symbolic evaluation: FunSearch and Alpha-Geometry pair LLM-generated solutions with external symbolic evaluators in Generate-Test-Critique loops.The external verifier is described as critical for avoiding flawed approximate solution candidates.
- Reinforcement learning: In reinforcement learning with simulators, simulators provide the evaluation and critique roles that external critics perform in LLM-Modulo frameworks.LLMs can also help produce approximate transition, reward, and high-level action models.
- Multimodal models: Multimodal LLMs expand System 1 imaginative coverage, but the paper does not establish that multimodality supplies System 2 competence.The paper identifies visual-language models as potential tools for criticizing robot-behavior style.
- Agentic systems: Agentic frameworks can invoke external services, but action invocation alone does not guarantee that the resulting actions achieve a desired state.The paper presents robust planning capabilities as necessary for such guarantees.
6. Conclusion
The paper concludes that LLMs cannot plan independently but can support robust planning as approximate knowledge sources and candidate generators within externally verified frameworks. It presents this position as a response to both excessive optimism and excessive pessimism about LLMs.
- Conclusion: LLMs cannot plan themselves, but can contribute approximate knowledge and candidate plans when combined with external sound model-based verifiers.The proposed framework integrates LLM idea generation with verifiers that provide correctness guarantees.
- Conclusion: The paper attributes many planning and verification claims to conflating approximate knowledge acquisition with generating executable plans of action.
- Conclusion: LLM-Modulo frameworks aim to integrate LLM capabilities with external verification for robust and expressive planning without inheriting traditional symbolic planners’ expressiveness and search-complexity limitations.
Impact Statement
The paper advocates caution about autonomous LLM planning while proposing externally supported LLM use for robust planning. It identifies potential significance for mission-critical situations given current interest in agentic LLMs.
- Impact Statement: The paper recommends caution against misapplying LLMs in autonomous planning modes.
- Impact Statement: LLM-Modulo frameworks are proposed as a way to leverage LLMs for robust planning.
- Impact Statement: These insights could have significant positive impact in mission-critical situations amid current interest in agentic LLMs.