PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning and Leveraging Verifiers to Improve Planning Capabilities of Pre-trained Language Models

Daman Arora, Subbarao Kambhampati

arXiv:2305.17077v1cs.CLcs.AI

TL;DR

Planning performance of pretrained language models remains questionable, and finetuned models often produce invalid actions by missing action preconditions. The paper trains a verifier from sampled negative actions to filter generated plans, reporting improved Blocksworld results and better verifier generalization when the generator itself is finetuned for verification.

  • Problem

    Language models show poor planning performance, and finetuned models often violate action preconditions, yielding invalid plans.

    Source: Introduction · p. 1

  • Method

    A finetuned GPT-2 generator proposes state transitions, while a verifier trained on valid transitions and randomly sampled actions classifies action applicability.

    Source: Verifier Augmented Plan Generation in… · p. 2, Training the Verifier · p. 2

  • Results

    On Blocksworld, the verifier-equipped generator gains 27% over the pure-generator baseline, and bad trajectories decrease from 52% to 5%.

    Source: Introduction · p. 1

  • Takeaways & Limitations

    Diverse trajectory sampling with verifier pruning is reported as critical for improving success, and finetuning the generator for verification generalizes better than finetuning vanilla GPT-2.

    Source: Introduction · p. 1

    The method resets to the initial state after a rejected transition; resetting instead to the previous state can leave an unsolvable state-goal problem even when the original problem is solvable.

    Source: Generating the training data · p. 2

Abstract

from arXiv · show

There have been wide spread claims in the literature about the emergent reasoning capabilities of Pretrained Large Language Models. However, recent studies, have found that their ability to plan remains questionable. Through our experiments using GPT-2, we empirically demonstrate that the performance of a finetuned baseline remains poor because it violates pre-conditions of actions in the plans that it generates. To improve the planning capabilities of a finetuned LLM, we train a verifier, which can classify actions as being valid or invalid in a particular state. By randomly sampling actions from the same dataset, we generate examples of invalid actions which are then used to train a verifier which can check for action applicability. In the presence of diverse sampling from a generator and a verifier which can prune invalid trajectories, we show significant gains in the success rate on the Blocksworld domain. Additionally, we show that finetuning the GPT-2 generator itself to create the verifier generalizes better than finetuning the base GPT-2. Lastly, we investigate the role of the sampling temperature which can be used to control the exploration-exploitation tradeoff.

Introduction

Although pretrained language models have shown reasoning abilities, their planning performance remains poor, including after finetuning. This paper links that weakness to invalid actions and evaluates verifier-guided planning on Blocksworld.

Introduction

  • Finetuned language models often violate action preconditions, producing invalid plans and poor test-instance success.The paper identifies this as a failure to capture action preconditions from trajectories.Source: Introduction · p. 1
  • The proposed verifier classifies actions as valid or invalid in a state, using randomly sampled dataset actions as inexpensive negative examples.The paper reports that diverse trajectory sampling combined with verifier pruning is critical to improving success rates.Source: Introduction · p. 1
  • Finetuned GPT-2 with a verifier gains 27% over the pure-generator baseline, while bad trajectories fall from 52% to 5%.The study also reports that finetuning the generator itself for verification generalizes better than finetuning vanilla GPT-2.Source: Introduction · p. 1

Verifier Augmented Plan Generation in LLMs

The paper uses a generate-test loop in which a finetuned GPT-2 proposes state transitions and another finetuned GPT-2 checks action applicability. The verifier is trained with valid transitions and randomly sampled actions as negative examples, and rejected transitions reset generation to the initial state.

Verifier Augmented Plan Generation in LLMs

  • PDDL actions are applicable when their preconditions hold in the current state, and their effects determine the next state.The framework represents planning problems through a domain, start state, and goal specification.Source: Verifier Augmented Plan Generation in… · p. 2

Training the Generator

  • The generator is finetuned on successful-trajectory transitions and generates an action and next state from the current state and goal.Training transitions pair the goal and current state with the action and resulting state.Source: Training the Generator · p. 2

Training the Verifier

  • Valid state-action pairs are positive verifier examples; pairing a state with a randomly sampled dataset action creates a negative example that may still be executable.The verifier is finetuned using the generator and predicts whether an action is applicable in the state.Source: Training the Verifier · p. 2

Verifier-augmented Generation

  • After each generated transition, the verifier checks its validity; accepted transitions continue from the next state, while rejected ones reset generation to the initial state and goal.The learned verifier can produce both false positives and false negatives.Source: Verifier-augmented Generation · p. 2

Experimental Evaluation

The experiments build a diverse offline dataset of valid Blocksworld trajectories and compare generation with and without verifier filtering. Planners are assessed by whether plans reach the goal and whether they contain an illegal action.

Generating the training data

  • Training data comes from randomized Blocksworld instances and a random-exploring agent; explored next states are pruned to create diverse valid trajectories.The setup generates 10,000 initial states and 200 test instances.Source: Generating the training data · pp. 2–3

Baselines and Metrics

  • The generator@k proposes a goal-reaching plan, while generator+verifier@k requires the verifier to approve every transition before proposing it.Both methods are evaluated with a maximum plan length; the baseline settings use top-p sampling with p = 0.99, temperature τ = 1, and Lmax = 40.Source: Baselines and Metrics · p. 3
  • Bad-transition-rate measures whether a proposed plan contains an illegal action, and goal-reaching-rate measures whether it achieves the goal.The paper notes that one inapplicable action makes the entire trajectory invalid.Source: Baselines and Metrics · p. 3

V : verifier,

The described inference procedure generates bounded plans and evaluates outcomes on 200 test instances. The experiments assume complete goal-state specifications, while partial specifications are described as an easy extension.

V : verifier,

  • The inference procedure takes a generator, maximum plan length, plan-attempt limit, start state, and goal as inputs, then generates action-state transitions within loops.The algorithm fragment returns an action prefix when the generated state matches the goal.Source: V : verifier · p. 3

How does generator+verifier do?

  • The evaluation uses k = 25 and runs the trained model on 200 test instances, with outcomes categorized by plan behavior.The supplied passage introduces three possible outcomes but does not enumerate them here.Source: How does generator+verifier do? · p. 3
  • The experiments use complete goal-state specifications; the authors describe partial goal specifications as an easy extension.Source: How does generator+verifier do? · p. 3

1. The proposed plan is valid and reaches the goal.5

On 200 Blocksworld test instances, generator+verifier@25 outperforms generator@25, and the verifier reduces bad transitions. The section also reports that verification scales with more attempts, generator-finetuned verifiers perform better, and temperature trades off exploration and exploitation.

1. The proposed plan is valid and reaches the goal.5

  • 65.5% of instances are solved by generator+verifier@25 versus 37.5% by generator@25, while bad transitions fall from 52.5% to 5%.Table 1 reports results on 200 Blocksworld test instances.Source: Table 1 · p. 3, §1 · p. 3

Does the performance improve with more attempts?

Finetuning the generator to train the verifier leads to better generalization.

How does the sampling temperature affect results?

  • The goal-reaching rate has an inverted U-shaped relationship with sampling temperature, reflecting an exploration-exploitation tradeoff.At τ = 0.2 the generator is conservative and makes fewer bad transitions; at τ = 1.1 it is more exploratory and makes more bad transitions.Source: How does the sampling temperature… · p. 4, Figure 3 · p. 4

Related Work

The paper situates its verifier among trajectory-based planning and offline learning approaches, distinguishing it from methods that rely on external simulators for correctness.

  • Unlike SayCan (Ahn et al. 2022), which uses LLMs for heuristic guidance and external simulators for correctness, this verifier is learned from the same trajectory data used to finetune the LLM.Source: Related Work · p. 4
  • The paper relates its approach to offline RL, where learning from system logs can reduce interactions required to learn environment dynamics or a policy.Source: Related Work · p. 4
  • Prior trajectory-based planning work includes RNN plan recognition, Decision Transformer policies, and LLMs trained on planner-generated action sequences, but these works do not use independently trained verifiers.Source: Related Work · p. 4

Conclusions and Future Work

The authors conclude that verifiers can prune plans with illegal actions and support scaling to more plan-generation attempts. They leave verifier feedback, larger pretrained models, and in-context verifier construction for future investigation.

Training data

The training data are generated from Blocksworld initial states by sampling valid actions into trajectories, then collecting transitions with the trajectory endpoint as the goal.

  • The training set starts from 10,000 initial states containing 3 to 8 blocks, with trajectories generated by sampling valid actions.Source: Training data · p. 5
  • Trajectories are capped at length 20 and stopped when a previously visited state recurs, yielding almost 61,000 trajectories.Source: Training data · p. 5
  • Each transition is collected with the trajectory’s end state as its goal, and a similar validation dataset contains 6,000 trajectories.PDDL states and actions are textualized by listing true propositions in lexicographical order.Source: Training data · p. 5

Test instances

Test trajectories comprise 200 runs on different initial states, each at most 30 steps, with goals drawn from the second half of a generated trajectory.

Test instances

  • 200 test trajectories use different initial states and are capped at 30 steps, with goals sampled from the second half of each trajectory.This makes reaching the sampled end state require a non-trivial number of actions.Source: Test instances · p. 5

Generator

This section describes the finetuned GPT-2 generator and verifier and illustrates generated planning behavior with a Blocksworld trajectory. It shows that generated plans need not be near-optimal, as the training trajectories are not near-optimal.

Generator

  • The generator is a finetuned GPT-2 model trained on 2 Nvidia V100 GPUs.Its training hyperparameters are listed in Table 3.Source: Generator · p. 5

Verifier

  • The verifier is finetuned on the GPT-2 generator and trained on 1 Nvidia V100 GPU.Its training hyperparameters are listed in Table 4.Source: Verifier · p. 5

An Example Plan

  • The example plan generated by generator+verifier@25 is not necessarily near-optimal because the training data does not use a near-optimal planner for trajectories.The passage states that plan quality would also improve if near-optimal trajectories were used for training.Source: An Example Plan · p. 5
  • The example records unstacking b2 from b3 and picking up b3, followed by a state in which b3 is on b4.These transitions are shown alongside the same goal configuration.Source: An Example Plan · pp. 9–10
Loading 2305.17077v1…