PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

Kaya Stechly, Karthik Valmeekam, Subbarao Kambhampati

arXiv:2402.08115v2cs.AI

TL;DR

LLMs’ ability to self-critique and improve reasoning solutions remains uncertain. This paper evaluates self-critique and external verification across three reasoning and planning domains, finding that self-verification worsens performance while sound external verification improves it.

  • Problem

    The paper investigates whether LLMs can reliably self-critique and improve solutions on reasoning and planning tasks.

    Source: §1 · p. 1

  • Method

    The paper evaluates GPT-4 iterative solution and verification systems, comparing self-critique with external sound verification across three reasoning and planning domains.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Self-verification worsens performance across almost all domains, whereas external sound verification yields substantial gains and repeated prompting preserves most benefits without critique.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Reasoning systems should rely on external sound verification rather than opaque self-critique when possible.

    Source: §1 · p. 2, §6 · p. 9

    Common evaluation domains may suffer from memorization, insufficient difficulty, and missing ground truth, limiting how informative or generalizable conclusions are.

    Source: §2 · p. 4

Abstract

from arXiv · show

There has been considerable divergence of opinion on the reasoning abilities of Large Language Models (LLMs). While the initial optimism that reasoning might emerge automatically with scale has been tempered thanks to a slew of counterexamples--ranging from multiplication to simple planning--there persists a wide spread belief that LLMs can self-critique and improve their own solutions in an iterative fashion. This belief seemingly rests on the assumption that verification of correctness should be easier than generation--a rather classical argument from computational complexity--which should be irrelevant to LLMs to the extent that what they are doing is approximate retrieval. In this paper, we set out to systematically investigate the effectiveness of iterative prompting in the context of reasoning and planning. We present a principled empirical study of the performance of GPT-4 in three domains: Game of 24, Graph Coloring, and STRIPS planning. We experiment both with the model critiquing its own answers and with an external correct reasoner verifying proposed solutions. In each case, we analyze whether the content of criticisms actually affects bottom line performance, and whether we can ablate elements of the augmented system without losing performance. We observe significant performance collapse with self-critique and significant performance gains with sound external verification. We also note that merely re-prompting with a sound verifier maintains most of the benefits of more involved setups.

1 Introduction

The paper investigates whether iterative self-critique improves LLM reasoning on formally verifiable tasks, finding that self-verification degrades performance while external sound verification produces substantial gains with simpler prompting.

Motivation

  • The work challenges optimism about LLM self-critique, which assumes verification is easier than generation despite evidence of weaknesses on arithmetic, logic [11], and planning [44].The authors argue that this complexity-based assumption may be irrelevant if LLM behavior is better modeled as approximate retrieval.Source: §1 · pp. 1–2

Methodology

  • The study evaluates GPT-4 self-critique on Game of 24, Graph Coloring, and STRIPS planning, selected because their formal correctness can be automatically checked.Formal tasks enable evaluation of both binary verification and generated critiques, unlike style-based qualitative tasks such as creative writing [52].Source: §1 · p. 2
  • The proposed LLM+LLM system uses GPT-4 [2] to generate a solution, verify or critique it, and feed back the result through iterative backprompts.The same model produces both candidate solutions and verification feedback.Source: §1 · p. 2

Results

  • Self-verification worsens performance across almost all domains because the verifier frequently rejects valid solutions, with more backprompts consistently degrading output quality.The analysis attributes the collapse to a significant false-negative rate in the verifier LLM.Source: §1 · p. 2
  • External sound verification substantially improves performance across Game of 24, Graph Coloring, and STRIPS planning, while repeated prompting without critique preserves most or all of these gains.The critique level often does not matter when the verifier is sound, and repeatedly querying the same base prompt until certification matches the benefits of more complex setups.Source: §1 · p. 2

Contribution

  • The findings support LLM-Modulo systems, in which external sound systems perform verification rather than relying on opaque self-critique.The paper warns that iterative prompting benefits can be misattributed to rich feedback or self-critique.Source: §1 · p. 2

2 Related Work

Prior work presents both strong claims that LLMs reason and self-correct and systematic evidence of brittle, domain-sensitive performance. This work narrows evaluation to formally verifiable reasoning tasks and addresses gaps in benchmark coverage, self-verification analysis, and evaluation validity.

Prior findings

  • Systematic studies report low scaling returns and brittle performance across reasoning domains, despite initially promising anecdotal and benchmark results [4] [25] [44] [11] [47] [52] [1] [42] [12] [32] [30] [3].Research spans basic reasoning, planning, logic, arithmetic, analogy, and math puzzles.Source: §2 · p. 2

Self-critique literature

  • Self-critique methods repeatedly claim that iterative feedback enables LLMs to learn from mistakes and improve without human feedback [37] [5] [29], but systematic studies find brittle gains and occasional degradation versus sound verification [16, 18, 14].The proposed mechanism is that self-reflective feedback supplies a semantic gradient or concrete direction for improvement [37].Source: §2 · p. 3

Contribution

  • This work extends prior studies by testing autonomous multi-round self-verification, up to 15 rounds, across three formally verifiable domains and thoroughly ablating the system to locate performance deterioration.The authors argue these domains provide broader and better tests of reasoning and self-correction capability.Source: §2 · p. 3

Scope of reasoning

  • Because reasoning claims use inconsistent definitions and assumptions, the study restricts evaluation to fully specified problems solvable by deductive methods, which it treats as fundamental components of reasoning rather than recall.The authors argue that other reasoning tasks must include these capabilities, even when combined with commonsense, language, or domain-specific components.Source: §2 · p. 3

Benchmark coverage

  • Previous work examined LLM planning and other classical reasoning problems [43] [44] [40] [11], but no previous work carefully covered a broad range of formal verification problems, and limited benchmark coverage can contribute to an illusion of greater LLM competence.Source: §2 · p. 4

Evaluation limitations

  • Common evaluation domains are weakened by test-set memorization, insufficient difficulty, and absent ground truth, making self-verification effects and reported conclusions difficult to establish [34, 51, 39, 18, 29, 52, 28, 49, 14].Multiple-choice formats can make refinement trivial, while human or model-based scoring may lack validated ground truth.Source: §2 · p. 4

3 Background On Test Tasks

The study evaluates GPT-4 self-critique across Game of 24, Graph Coloring, and STRIPS planning, selected for flexible generation, formal verifiability, and reduced reliance on memorized patterns. Each task also supports sound external verification of proposed solutions.

3 Background On Test Tasks

  • The three tasks provide many possible solutions, formal verification, and arbitrary new instances, reducing the chance that pattern matching or approximate retrieval substitutes for reasoning.This design also ensures that removing a few candidate solutions does not trivialize iterative self-verification, unlike multiple-choice settings.Source: §3 · p. 4

3.1 Game of 24

  • Game of 24 requires combining four numbers with basic arithmetic and parentheses to produce 24, using 4nums.com instances for comparison with prior work [52].Generation uses instances 901–1000, while verification and critique use instances 1–1000; SymPy3 checks expressions and flags malformed outputs.Source: §3.1 · pp. 4–5

3.2 Graph Coloring

  • Graph Coloring uses planar graphs with known optimal color counts, requiring distinct colors on every edge; its sound verifier checks each edge with a simple Python loop.The test set contains 100 small Erdős–Rényi graphs with p = 0.4, formatted in DIMACS and paired with precalculated chromatic numbers.Source: §3.2 · p. 5

3.3 STRIPS planning

  • STRIPS planning asks for action sequences that transform an initial state into a goal state while satisfying preconditions, evaluated in Blocksworld and Mystery Blocksworld domains.Plans are verified by executing actions in order and checking preconditions and goal satisfaction with the sound external critique VAL.Source: §3.3 · pp. 5–6

4 Methodology

The methodology evaluates interchangeable prompting architectures for iterative self-critique, using complete interaction histories and domain-specific components to separate guessing, verification, and critique roles. It compares LLM-based verification and critique with provably sound verification while progressively reducing feedback information.

4 Methodology

  • The framework sends a translated problem to an LLM, critiques its proposal in a separate query, and iterates by appending extracted feedback and the full interaction history until acceptance or timeout.The LLM acts as answer guesser, binary verifier, and critique generator within the loop.Source: §4 · p. 6
  • The study first measures complete-system performance and false-positive and false-negative verification rates, then replaces verification and critique with a provably sound verifier providing rich, correct feedback.These variations isolate the contributions of guessing, verification, and critique, including the LLM’s ability to implement feedback.Source: §4 · p. 6
  • Feedback is systematically reduced from task-specific evaluations to “the previous answer was wrong” and finally to re-querying with the base prompt, while tracking whole-system performance.This ablation tests how much information the iterative loop needs to retain its effects.Source: §4 · p. 6
  • LLM critique generation is evaluated on domain subproblems by comparing parsed critiques against a sound verifier’s output for proposed solutions.The prompts include a problem description and candidate solution and request domain-specific criticism when incorrect.Source: §4 · p. 6
  • Sound verifiers provide task-specific feedback, including arithmetic evaluation, graph-coloring constraint violations, and planning precondition or goal-failure explanations.These critiques supply concrete correctness information tailored to each domain.Source: §4 · p. 6

5 Examining Self-Verification

Across 100 instances per domain, self-critique reduces performance, whereas sound verification substantially improves it; these gains largely persist without critique content. LLM-generated verification and critiques contain errors that can compound and drive self-critique below standard prompting.

5 Examining Self-Verification

  • LLM verifiers’ high false-negative rates reject many answers and cause later, worse generations, making them mostly insufficient for reliable reasoning verification.Game of 24 and Blocksworld fare better than the other domains, with Blocksworld showing a modest but still inferior improvement relative to sound verification.Source: §5 · pp. 7–8, Table 2 · p. 8
  • Sound verification substantially outperforms standard prompting across domains, while self-critique decreases performance and can collapse as backprompts increase.With a sound verifier, remaining errors arise from the generator failing to produce a correct answer before timeout; with self-verification, incorrect rejections can lead to worse later generations.Source: §5 · pp. 6–8, Figure 2 · p. 7

5.1 Critique generation

  • In Game of 24, the LLM labels 79.1% of guaranteed-correct equations as correct but evaluates 81.6% as equal to 24.Thus, it sometimes evaluates an equation correctly while still marking it wrong.Source: §5.1 · p. 8
  • Graph Coloring critiques contain non-existent edges and spurious vertex-color claims, while planning critiques hallucinate whether action preconditions hold.Mystery Blocksworld critiques also incorrectly assume precondition states, lowering critique accuracy.Source: §5.1 · p. 8
  • LLM critiques introduce verification errors and misleading feedback, compounding until self-critique performs below simply taking the model’s first guess.Errors include passing over correct answers, accepting wrong ones, and biasing future outputs away from correct answers.Source: §5.1 · p. 8

5.2 Critique consideration

  • Providing binary, first-error, or all-errors feedback produces little performance difference, and increasing feedback decreases performance in two domains.The intended advantage of detailed feedback is therefore not consistently observed.Source: §5.2 · pp. 8–9
  • Repeated sampling with a sound verifier achieves comparable gains to feedback conditions, reduces prompt token cost quadratically, and self-consistency shows no improvement over standard prompting.The results indicate that performance gains stem largely from enough guesses plus sound verification rather than critique information.Source: §5.2 · p. 9

6 Conclusion

Across three reasoning and planning tasks, LLMs performed poorly at verification, critique generation, and critique consideration, while external sound verification improved performance. The authors therefore recommend repeated solution proposals paired with sound or partial verification signals.

6 Conclusion

  • LLMs performed poorly in verification, critique generation, and critique consideration, often making self-critique worse than an upfront guess.The stacked errors in these roles caused self-critiquing loops to underperform direct solution guessing across the evaluated hard domains.Source: §6 · p. 9
  • External sound verification improved performance, and repeated guesses with a sound verifier could work without generating critiques.The verifier can select any correct proposal from multiple LLM guesses.Source: §6 · p. 9
  • These results contradict earlier optimism about LLM self-critique and extend studies limited by easy benchmarks, unclear ground truth, or test-set memorization.The paper positions its findings as a correction and deeper evaluation of that prior literature.Source: §6 · p. 9
  • The authors recommend embedding LLMs in systems that generate multiple guesses and provide a signal indicating when a solution is good enough.Suggested signals include VAL for STRIPS planning, expression simplification for Game of 24, and constraint checking for constraint satisfaction problems.Source: §6 · p. 9
  • For real-world applications, verification may combine partial critics with consensus over criteria that each critic can assess.The authors connect this architecture to prior success and the LLM-Modulo framework [23].Source: §6 · p. 9

A Appendix · A.1 Prompt Variation and Chain of Thought

Prompt and chain-of-thought variations only partly improve verification, and their gains remain below sound-verifier performance while substantially increasing cost. Effects also vary across domains, with limited improvements on tasks where verification is simple and linear.

A.1 Prompt Variation and Chain of Thought

  • CoT verification improves performance in some, but not all, prompting experiments.Table A1 compares the full pipeline using CoT verification with the main-paper results across 100 instances per domain.Source: §A.1 · p. 14, Table A1 · p. 14
  • CoT raises Game of 24 verification accuracy from 87% to 99%, but the full pipeline still trails the sound verifier by 6 percentage points.The improvement also requires 17 times more output tokens, more than doubling verification cost.Source: §A.1 · p. 14, Table A1 · p. 14
  • CoT effectiveness varies substantially by domain, and prior work finds it fails to generalize robustly in reasoning domains such as Blocksworld [40].The appendix therefore cautions against treating CoT as uniformly effective across tasks.Source: §A.1 · p. 14
  • On tasks with relatively simple and linear verification, CoT produces far smaller gains than in Game of 24.These tasks are theoretically amenable to CoT, yet the observed improvements are not nearly as significant.Source: §A.1 · p. 14
  • The study reruns verification-only experiments with CoT and altered non-CoT prompts to test prompt sensitivity.Table A2 reports these results alongside the original experiments, including answer-reasoning order swaps and CoT cases.Source: §A.1 · p. 14, Table A2 · p. 14
  • The appendix evaluates GPT-4’s Game of 24 assessment and verification behavior across different expression types.Table A3 presents the comparison by expression type.Source: Table A3 · p. 15

A.2 On Tree of Thoughts

On the same Game of 24 test set, 150 direct non-CoT queries with a sound verifier reached 70%, comparable to Tree of Thoughts’ reported 74% at similar cost. The remaining difference may reflect ToT’s breadth-first search over easier intermediate steps and heuristic evaluations rather than full solutions and reflections.

Direct comparison

  • 70% with 150 direct non-CoT queries and a sound verifier was comparable to Tree of Thoughts’ reported 74% at roughly equivalent cost.The verifier used 150 queries, while ToT averaged the cost of about 150 Chain-of-Thought prompts.Source: §A.2 · p. 15

Interpretation

  • ToT’s classical breadth-first search prompts the LLM for intermediate steps and heuristic evaluations rather than full solutions and reflections.This reduces compositionality by offloading search structure to a proven classical algorithm.Source: §A.2 · p. 15
  • The paper argues that these techniques may not scale beyond the simplest toy instances because they sidestep major hurdles in LLM reasoning.The comparison addresses an apparent discrepancy with [52] by using the same test set and comparable query cost.Source: §A.2 · p. 15

A.3 Game of 24 · A.3.1 Evaluation vs. Binary Verification for Game of 24

This section compares GPT-4’s evaluation and binary-verification abilities on Game of 24 by testing proposed expressions with and without the goal state. It evaluates multiple expression-error conditions and reports results across a broad sample, with fewer LLM-generated cases.

A.3.1 Evaluation vs. Binary Verification for Game of 24

  • The experiment compares GPT-4’s critique and verification abilities on Game of 24.Source: §A.3.1 · p. 15
  • Each query asked GPT-4 to evaluate an expression and determine whether it equals 24.Source: §A.3.1 · p. 15
  • Proposed expressions comprised correct, ablated-operation, ablated-number, random, and LLM-generated cases.The ablated-operation and ablated-number cases each contained exactly one wrong operation or number, respectively.Source: §A.3.1 · p. 15
  • Additional no-info cases requested expression evaluation without stating the goal state or asking for verification.These cases included one correct and one random expression condition.Source: §A.3.1 · p. 15
  • The experiment generated 1000 expressions for each non-LLM type, covering every problem in the full set.Source: §A.3.1 · p. 15
  • The LLM condition used 100 expressions because generations were restricted to instances 901–1000 in the main paper.Table A.3.1 summarizes the resulting comparison.Source: §A.3.1 · p. 15

A.3.2 Prompts

The prompts specify a Game of 24 task requiring valid arithmetic expressions and structured outputs. Verification variants ask the model to assess expressions directly or with step-by-step reasoning and JSON feedback.

Base task

  • The base prompt asks the model to use the given numbers and basic arithmetic operations to obtain 24, then place its answer before an [ANSWER END] tag.The example input is 1 1 4 6.Source: §A.3.2 · p. 16

Verification

  • The direct-verification prompt asks whether an expression uses only the given numbers and evaluates to 24, requiring JSON fields for the evaluated number and correctness.It also restricts the response to parseable JSON with no additional text.Source: §A.3.2 · p. 16

CoT verification

  • The chain-of-thought verification prompt requires checking number usage, multiplicities, and arithmetic correctness step by step before returning tagged JSON feedback and a correctness judgment.If the expression is incorrect, the model must explain how to correct it.Source: §A.3.2 · p. 16

A.3.3 Backprompts

The backprompt variants ask the model to retry after feedback from self-critique or sound verification, including malformed-answer handling and exact-number constraints.

Sound verifier

  • Sound-verifier backprompts request a new expression after rejecting an answer as incorrect, while preserving the required numbers, target value, and answer format.The pass/fail variant gives concise feedback and the stronger variant identifies misuse of the required number multiset.Source: §A.3.3 · p. 17

Malformed answer handling

  • The verifier labels the submitted expression malformed and requests a correct expression using 1, 1, 4, and 6 that evaluates to 24, followed by [ANSWER END].The malformed answer is ((6 / (1 - 1/4)) = 24[ANSWER END].Source: §A.3.3 · p. 17

A.3.4 Complete Example of Iterative Prompt Sequence

The example shows a seven-backprompt iterative sequence generated by a sound verifier, with feedback repeatedly identifying errors until successful task completion. The interaction progressively diagnoses malformed expressions, invalid arithmetic, and incorrect number usage in the Game of 24 task.

Iterative prompt sequence

  • A sound verifier generates seven successive backprompts, stopping when the Game of 24 task is successfully completed.The sequence is explicitly labeled as stopping at successful completion, although the supplied excerpt does not show the final completed expression.Source: §A.3.4 · p. 17
  • Verifier feedback first flags malformed expressions and division by zero before requesting another expression using the same four numbers.The sequence rejects malformed forms such as expressions containing 1 − 1 and identifies the resulting division by zero.Source: §A.3.4 · p. 17
  • Later feedback checks exact number usage and evaluates proposed expressions, rejecting solutions that omit a required 1 or compute 15, 18, or 17 instead of 24.The verifier explicitly requires exactly [’1’, ’1’, ’4’, ’6’] and reports the incorrect evaluated results.Source: §A.3.4 · p. 18

A.4 Graph Coloring … A.5.9 LLM as Verifier - Mystery Blocksworld

This section examines GPT-4’s verification and iterative backprompting across graph coloring, Blocksworld, and Mystery Blocksworld, detailing the prompts, feedback, and evaluation procedures used. It presents examples of self-generated and sound-verifier feedback, plan and coloring revisions, and the validity of resulting diagnoses and solutions.

A.4.1 Hallucinations in Graph Coloring Critique

  • The evaluated LLM more often hallucinated edges than vertex colors, typically claiming same-colored nonadjacent vertices were connected, while hallucinations decreased proportionally as domain error increased.It was queried on 500 proposed colorings spanning correct, ablated one-edge-wrong, non-optimal, random, and LLM-generated types.Source: §A.4.1 · pp. 18–19
  • The critique outputs also included rare question reversals and self-contradictions, such as asserting that same-colored vertices violated constraints because they were not connected.The overlap between edge and vertex hallucinations was rarer than either type individually, and the model did not invent new vertex names.Source: §A.4.1 · p. 19

A.4.2 Prompts

  • The prompts required complete color assignments under a three-color limit and asked the verifier to report whether a coloring was correct or identify same-color adjacent vertices.Alternative chain-of-thought prompts additionally requested structured fields for wrong edges, vertex coverage, optimality, and correctness.Source: §A.4.2 · pp. 21–22

A.4.3 Backprompts

  • Backprompts converted verifier feedback into requests for a corrected coloring, ranging from a pass/fail message to first-error and full critiques.Examples identify vertices 9 and 11 as sharing Color3 across an edge, while the full critique also flags vertices 10–13 and 11–13.Source: §A.4.3 · pp. 23–24

A.4.4 Complete Example of Iterative Prompt Sequence

  • A complete example presents a sequence of 15 LLM-generated backprompts beginning from the graph-coloring task and its formatting requirements.The excerpt includes the graph, a three-color constraint, and instructions to label all 14 vertices.Source: §A.4.4 · p. 24
  • The graph-coloring example repeatedly supplies detected same-color edge conflicts and asks the model to regenerate a complete vertex-by-vertex coloring.Across successive attempts, feedback identifies conflicts such as vertices sharing an edge while having the same color, followed by another formatted coloring request.Source: §A.4.4 · p. 30

A.5.1 Accuracy of Planning Critique

  • LLMs often identify whether a proposed plan is valid but fail to generate the correct critique for invalid plans, including the invalidity type or valid explanation.The evaluation uses correct, inexecutable, non-goal-reaching, random, and LLM-generated plans over 100 instances, comparing binary verification and critiques with VAL.Source: §A.5.1 · p. 31

A.5.2 Prompts - Blocksworld

  • The Blocksworld prompts define pickup, unstacking, putting down, and stacking actions with preconditions concerning hand state, block position, and block clearance.They also require unstacking only when the stated block relation holds and permit stacking only while holding the block.Source: §A.5.2 · pp. 31–32
  • The Blocksworld prompts elicit verification through open-ended responses, fixed-format JSON, chain-of-thought reasoning, or reordered answer-and-reasoning outputs.The JSON formats distinguish invalid executable plans from goal failures using unmet preconditions or unmet goals.Source: §A.5.2 · pp. 33, 35, 36

A.5.3 Backprompts - Blocksworld

  • Self-critique incorrectly attributes a plan failure to the yellow block’s lack of clearance after the blue block is stacked on the red block.The sound verifier instead identifies step 7’s unmet precondition: the red block is clear.Source: §A.5.3 · p. 37
  • The full sound-verifier backprompt lists unmet preconditions at steps 7, 8, and 11, involving the red block, orange block, and yellow block respectively.The supplied passage presents this as the full verifier-generated feedback sequence.Source: §A.5.3 · p. 37

A.5.4 Complete Example of Iterative Prompt Sequence - Blocksworld

  • A complete example of the iterative prompt sequence is introduced for Blocksworld, beginning with the task description and block-arrangement setup.The supplied excerpt does not include the remainder of that sequence.Source: §A.5.4 · p. 37
  • GPT-4 corrected a Blocksworld plan after identifying that stacking the blue block made the yellow block non-clear, then produced a valid plan.The verifier first rejected the plan because the yellow block was not clear, and later accepted the corrected plan.Source: §A.5.4 · p. 39

A.5.5 LLM as Verifier - Blocksworld

  • In a separate Blocksworld example, GPT-4 as verifier incorrectly rejected a plan by claiming the yellow block was not clear after the blue block was stacked.The reported verifier response says the plan is invalid because the yellow block supposedly has the blue block on top of it.Source: §A.5.5 · p. 40

A.5.6 Prompts - Mystery Blocksworld

  • Mystery Blocksworld replaces ordinary block actions with Attack, Feast, Succumb, and Overcome, specifying each action’s required, added, and removed facts.The prompts provide initial conditions, goals, candidate plans, and both open-ended and format-based instructions for verifying validity.Source: §A.5.6 · pp. 40–41
  • Mystery Blocksworld verification prompts require JSON validity judgments and distinguish inexecutable plans from executable plans that fail their goals.The chain-of-thought variant additionally requests intermediate reasoning before the JSON response, while the swapped-order variant requests the JSON without that reasoning.Source: §A.5.6 · pp. 43–45

A.5.7 Backprompts - Mystery Blocksworld

  • The full sound-verifier backprompt identifies an invalid plan, an unmet goal, and unmet preconditions for actions at steps 3 and 5.It reports unmet goal condition object d craves object a, plus missing province object a for attack object a and province object d for feast object d from object a.Source: §A.5.7 · p. 46

A.5.8 Complete Example of Iterative Prompt Sequence - Mystery Blocksworld

  • In the complete iterative example, GPT-4 as verifier rejects a planner-generated sequence because its first Feast action lacks Province object b, after which the planner apologizes and proposes correction.The example presents a planner response followed by a verifier response and a subsequent planner revision.Source: §A.5.8 · p. 47
  • The verifier also detected later-step failures: after a valid Attack, Succumb could be invalid because its required state or object identity was not satisfied, preventing validation of the remaining plan.One example states that Attack makes Pain true but Province false, while another says Succumb must target the same object that was attacked.Source: §A.5.8 · p. 49
  • Repeated planner revisions remained invalid because proposed first actions violated Feast or Attack preconditions, including incorrect craving relations or mismatched Province and Planet objects.The verifier rejected plans when the initial action was impossible, and the planner repeatedly responded by revising the plan without producing a valid one.Source: §A.5.8 · pp. 49, 51, 52

A.5.9 LLM as Verifier - Mystery Blocksworld

  • GPT-4-as-Verifier explained invalidity by checking action preconditions, rejecting Feast when the acting object lacked the required Province condition and rejecting a later Succumb action when it targeted the wrong object.The examples explicitly enumerate Feast requirements and identify the mismatched object in the Succumb step.Source: §A.5.9 · p. 53

Related papers

Loading 2402.08115v2…