Source-linked AI summary
Chain of Thoughtlessness? An Analysis of CoT in Planning
Kaya Stechly, Karthik Valmeekam, Subbarao Kambhampati
TL;DR
The paper asks whether chain-of-thought demonstrations teach LLMs general reasoning procedures beyond examples. It evaluates prompt specificity and problem complexity in Blocksworld and scalable synthetic tasks, finding that gains require highly similar examples and quickly disappear as problems grow.
Problem
LLM reasoning performance often fails to generalize out of distribution, raising whether CoT demonstrations can teach procedures applicable beyond syntactically similar examples.
Method
The paper systematically evaluates CoT prompts across specificity levels and query complexities in Blocksworld, then tests scalable variants of three synthetic CoT tasks.
Results
CoT improves performance only when examples and queries are sufficiently similar; accuracy deteriorates with larger goal stacks and comparable failures appear on scalable synthetic tasks.
Takeaways & Limitations
The findings suggest CoT gains depend on prompt-specific pattern matching rather than in-context learning of general algorithms, requiring substantial human effort for specific prompts.
Takeaways & Limitations
The main planning case study focuses on simple table-to-stack Blocksworld problems, and observed improvements are modest and quickly deteriorate on slightly larger problems.
Abstract
from arXiv · showhide
Large language model (LLM) performance on reasoning problems typically does not generalize out of distribution. Previous work has claimed that this can be mitigated with chain of thought prompting-a method of demonstrating solution procedures-with the intuition that it is possible to in-context teach an LLM an algorithm for solving the problem. This paper presents a case study of chain of thought on problems from Blocksworld, a classical planning domain, and examines the performance of two state-of-the-art LLMs across two axes: generality of examples given in prompt, and complexity of problems queried with each prompt. While our problems are very simple, we only find meaningful performance improvements from chain of thought prompts when those prompts are exceedingly specific to their problem class, and that those improvements quickly deteriorate as the size n of the query-specified stack grows past the size of stacks shown in the examples. We also create scalable variants of three domains commonly studied in previous CoT papers and demonstrate the existence of similar failure modes. Our results hint that, contrary to previous claims in the literature, CoT's performance improvements do not stem from the model learning general algorithmic procedures via demonstrations but depend on carefully engineering highly problem specific prompts. This spotlights drawbacks of chain of thought, especially the sharp tradeoff between possible performance gains and the amount of human labor necessary to generate examples with correct reasoning traces.
1 Introduction
The paper studies whether chain-of-thought demonstrations teach LLMs general procedures for reasoning and planning. Across Blocksworld and scalable synthetic tasks, gains are narrow, deteriorate with problem complexity, and depend on highly specific examples.
- Chain-of-thought prompting is intended to teach procedures through annotated intermediate reasoning, but its generality and human-crafting requirements remain unclear.
- The study evaluates prompt generality and query complexity using Blocksworld, including scalable table-to-stack instances with goal stacks of height 3≤n≤20 and examples of height 2 and 3.
- Meaningful improvements occur only when hand-annotated examples and queries are sufficiently similar, while accuracy drops drastically as goal stack size increases.
- Increasing prompt generality reduces performance even on the smallest goal stacks, sometimes falling below standard prompting; self-consistency shows similar or worse behavior.
- The findings suggest that CoT gains reflect prompt-specific pattern matching rather than learning general algorithms, increasing the human effort needed to craft effective examples.
- Scalable Coin Flip, Last Letter Concatenation, and multi-step arithmetic tasks show similar lack of generalization when tested with GPT-4-Turbo.
2 Related Work
Related work presents CoT as a way to elicit reasoning through demonstrations, while prior analyses document brittleness and motivate testing out-of-domain generalization. This paper extends that concern by varying prompt specificity and scalable task difficulty.
- CoT builds on scratchpads by inserting human-crafted intermediate reasoning into examples, with prior work claiming improved reasoning without retraining.
- Consistency-oriented methods such as self-consistency, debate, least-to-most prompting, and verification can require multiple model calls and increase inference cost.
- Known limitations include loose correspondence between chains and answers, weak effects of annotation correctness, unfaithful generated reasoning, and sensitivity to example presentation.
- Prior work found limited out-of-domain generalization but did not vary prompt specificity, motivating the paper’s systematic comparison of specificity levels.
- The paper frames robust generalization on Blocksworld and scaled prior benchmarks as an expectation if CoT teaches compositional reasoning rather than pattern matching.
3 Background
Classical planning seeks action sequences that transform an initial state into a desired goal state, represented here with PDDL. The authors validate generated plans by checking execution and goal satisfaction.
- Classical planning asks a planner to find actions that move an agent from a specified initial state to a desired goal state, using the Blocksworld PDDL domain.
- PDDL separates a fixed domain of predicates, actions, preconditions, and effects from each problem’s initial state and goal expression.
- A valid plan satisfies each action’s preconditions during execution and reaches a goal-satisfying final state.
- The study converts natural-language model responses into PDDL and evaluates them with VAL.
4 Chain of Thought Setups for Planning
The planning experiments compare CoT prompts spanning universal, PDDL-specific, domain-specific, narrow table-to-stack, and lexicographic problem classes. Greater specificity supplies more targeted procedural knowledge but narrows the expected target distribution.
- Prompt specificity trades broader applicability and lower crafting effort against potentially higher performance from more granular task knowledge.
- CoT demonstrations are human-crafted thoughts intended to follow an algorithmic procedure for solving the example problems.
- The paper distinguishes the problems a designer expects an algorithm to solve from the narrower distribution on which the prompt actually gives robust performance.
- The study compares prompts from zero-shot CoT and general progression proofs to Blocksworld, table-to-stack, and lexicographic stacking procedures.
- A stacking prompt restricts examples to blocks initially on the table with a single specified goal stack, enabling near-direct pattern matching but making broad prompt coverage infeasible.
- Lexicographic stacking narrows the target further to goals with lexicographic prefixes such as A, AB, and ABC.
5 Blocksworld Results
Across Blocksworld evaluations, chain-of-thought gains were narrow: performance improved mainly for highly specific prompts and failed to generalize reliably to larger or broader problems.
- 5 Blocksworld Results: Chain of thought does not meaningfully enhance performance except on the narrowest problem distributions, where basic pattern matching can solve the task.The most successful procedure repeatedly stacks the next required block and stops when that block is unavailable.
- 5.1 Testing on Intended Problem Distributions: CoT fails to generalize beyond a handful of blocks, whereas sound planning systems achieve 100% accuracy on all tested problems.
- 5.2 Testing only on Table-to-Stack: On 261 table-to-stack instances, GPT-4 accuracy rises from 3.8% zero-shot to 59.3% with the stacking CoT prompt.
- 5.2 Testing only on Table-to-Stack: Only the most specific, least applicable prompt preserves much of the table-to-stack improvement, while other granularities lack robust stack-height generalizability.Self-consistency also reduced performance on these prompts.
- 5 Blocksworld Results: These results show a tradeoff: increasingly specific reasoning traces are harder to provide, yet modest gains quickly deteriorate when queried problems become slightly larger.
6 Extension to Scalable Synthetic Benchmarks
The authors extend CoT evaluation to scalable synthetic benchmarks and find that apparent gains generally fail to generalize as reasoning length or task variation increases.
- 6.1 Results: Only CoinFlip shows hints of length generalization: performance is perfect through 4 steps and falls below 90% only at 31-step problems.The task remains simple because it requires little beyond counting to roughly half the number of steps.
- 6.1 Results: CoT improves accuracy on small LastLetterConcatenation and arithmetic instances, but the gains quickly degrade as the number of required steps increases.The same pattern appears across the scalable synthetic benchmarks.
- 6.1 Results: In LastLetterConcatenation, CoT generalizes syntactic properties such as containing the correct letters, but not their exact order or number.Overall accuracy returns to direct-prompting performance despite improvements in Levenshtein distance and letter inclusion.
- 6.1 Results: Across letter-concatenation and arithmetic variants, CoT gains persist longer on easier cases, suggesting a difficulty-dependent sweet spot before performance collapses.Prompt granularity is harder to study in these simpler domains because they offer less variation.
- 6.1 Results: Exhaustive tests show every evaluated model achieves 100% on single-digit intermediate arithmetic operations, yet CoT still fails to generalize to arbitrary lengths.This rules out accumulating intermediate-function error as the explanation within this task and points instead to failure to learn the correct algorithm from demonstrations.
- GPT-4-Turbo shows similar generalization failures on scalable CoinFlip, LastLetterConcatenation, and multi-step arithmetic tasks as in Blocksworld.These failures become visible when problems require more reasoning steps or when minor domain modifications are tested.
7 Conclusion
The paper concludes that CoT demonstrations do not reliably induce general algorithmic procedures in current state-of-the-art LLMs. Gains are strongest for narrow, highly specific problem classes, but require substantial human effort and disappear when queries differ in generality.
- Systematic evaluation on classical planning finds that CoT examples do not induce a general ability to apply the demonstrated procedure to novel instances.Performance improvements quickly vanish when queries differ in generality from the examples, even when the same algorithm applies.
- Very specific prompts are more likely to improve performance, but crafting them can require significantly more human labor.The paper therefore identifies a tradeoff between prompt specificity and the effort needed to produce useful reasoning traces.
- The results suggest CoT may work consistently only within sufficiently narrow problem classes, with pattern matching potentially explaining gains better than general procedure learning.This conclusion is presented as contrary to stronger claims that demonstrations teach broadly applicable algorithms.
A.1 Broader Impacts
The study challenges the prevalent belief that LLMs can operationalize and generalize chain-of-thought advice. It warns that CoT may encourage false confidence on planning and reasoning tasks.
- The study questions whether LLMs can operationalize and generalize chain-of-thought advice effectively.This challenges a prevalent assumption behind widespread CoT use for planning and reasoning.
- The authors suggest CoT works mainly when models can straightforwardly pattern-match examples to problems.This implication limits the expected transfer of demonstrated procedures beyond closely matching instances.
- These findings may help mitigate false confidence when applying LLMs to tasks requiring planning and reasoning.
A.2 Self Consistency on Table to Stack problems
Self-consistency does not produce a generalization breakthrough on table-to-stack Blocksworld. Across the tested instances, it is generally worse than the original CoT results.
- Self-consistency generally performs worse than the original table-to-stack CoT results.The evaluation sampled five reasoning paths at temperature 0.7 and selected the most frequent plan, breaking ties randomly.
- Sampling five reasoning paths does not create a generalization breakthrough for planning problems.Most queries produced five unique responses, limiting the usefulness of frequency-based selection.
- The authors attribute this weakness partly to planning's larger solution space than that of many earlier multiple-choice benchmarks.
A.3 Further details on modifications to the CoinFlip domain
The modified CoinFlip evaluation scales generated instances by increasing the number of names. It includes a 1,120-instance main set and an extended test showing stronger degradation beyond 30 names.
- The main CoinFlip test set contains 1,120 instances spanning 1 to 28 names, with 40 instances per name count.Instances are generated by filling a template with names sourced from the U.S. Social Security Administration and filtered to at least 50 occurrences.
- Figure A.1.1 compares GPT-4-Turbo, Claude-3-Opus, and GPT-4 across CoT prompting methods as block count increases.
- Table 3 reports self-consistency accuracy over 261 table-to-stack Blocksworld instances.
- An extended 2,960-instance evaluation reaches 75 names and shows performance decreasing more significantly past 30 names.
A.4 Further details on modifications to the LastLetterConcatenation domain
The modified LastLetterConcatenation domain is scaled by increasing the number of words whose final letters must be concatenated. The evaluation covers 800 problems from 1 to 20 words.
- The LastLetterConcatenation problems scale by increasing the number of words whose last letters must be concatenated.
- The evaluation contains 800 problems, with 40 instances for each word count from 1 through 20.
A.5 Further details on the multi-step Arithmetic dataset
The multi-step arithmetic dataset contains problems spanning 1 to 29 operations, with 40 instances at each operation count, and compares general versus digit-constrained prompting.
- Dataset construction: 1 to 29 operations are represented, with 40 arithmetic problems for each operation count.The test set contains 1160 total problems.
- Related visual: The supplied figure is labeled as reporting table-to-stack accuracy for GPT-4-Turbo and Claude-3-Opus across CoT methods with self-consistency.
- Prompt variants: The experiments compare prompting for general expression simplification against prompting that requires every intermediate step to be a single digit.
A.6 Planning Prompts and Responses by GPT-4
This appendix illustrates Blocksworld action semantics and GPT-4 prompting examples, including a universal algorithm that clears towers before constructing goal stacks.
- A.6.1 Domain Information: Blocksworld actions are pick up, unstack, put down, and stack, governed by hand, location, and clear-block preconditions.Stacking makes the supporting block no longer clear, while putting down or stacking empties the hand.
- A.6.3 Blocksworld Universal Algorithm Prompt: The prompt includes explicit thought traces, state descriptions, goals, and a required plan format before presenting a new problem.
- A.6.3 Blocksworld Universal Algorithm Prompt: After clearing the table, the algorithm identifies tower bases from the goal and stacks each required tower from its base upward.The examples select the only block that need not sit on another block as the tower base, then place successors in goal order.
- A.6.3 Blocksworld Universal Algorithm Prompt: The universal algorithm first dismantles existing towers onto the table, postponing blocked moves until the obstructing blocks become clear.It then verifies that all blocks are clear and on the table.
- A.6.3 Blocksworld Universal Algorithm Prompt: The worked universal-prompt example constructs a tower by stacking Block C on Block D, Block A on Block C, and Block B on Block A.
A.6.2 Progression Proof Prompt
The progression-proof prompt teaches GPT-4 to produce plans with explicit state transitions, action justifications, and goal verification across multiple Blocksworld examples.
- A.6.2 Progression Proof Prompt: Each progression-proof example records the current state, selected action, precondition-based reason, and resulting state.The prompt is designed around stepwise state progression rather than an abstract plan alone.
- A.6.2 Progression Proof Prompt: The demonstrations include multiple example prompts and plan formats, including an unstacking action whose applicability is justified by block clearance and hand emptiness.
- A.6.2 Progression Proof Prompt: The demonstrations cover simple table-to-stack construction, including picking up a clear table block and stacking it onto a clear target.
- A.6.2 Progression Proof Prompt: The examples explicitly conclude that the plan is valid once the goal conditions hold in the final state.
- A.6.2 Progression Proof Prompt: The prompt also demonstrates rearranging an existing tower by unstacking clear blocks, putting them down, and rebuilding the requested arrangement.
- NeurIPS Paper Checklist: The appendix checklist materials state that the paper reports no error bars because they would be too expensive and marks broader impacts as not applicable.