PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Script Knowledge from Large Language Models for Constrained Language Planning

Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Robert Jankowski, Yanghua Xiao, Deqing Yang

arXiv:2305.05252v5cs.CLcs.AI

TL;DR

Planning research has largely focused on abstract stereotypical goals, leaving specific goals with constraints understudied. The paper defines constrained language planning, uses over-generate-then-filter to improve LLM planning and distills CoScript. It reports improved LLM planning and strong performance from smaller models trained on CoScript.

  • Problem

    Prior work mainly plans for abstract stereotypical goals, while planning for specific goals with multiple facets of constraints remains understudied.

    Source: §1 · p. 2

  • Method

    The paper uses over-generate-then-filter to improve LLM scripts for constrained goals and distills the resulting knowledge into CoScript, a 55,000-script dataset.

    Source: §1 · p. 2

  • Results

    The method improves constrained planning quality for LLMs, and smaller models trained on CoScript can outperform LLMs.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    CoScript is presented as a resource for advancing language planning research on more complex and diverse goals and constraints.

    Source: §7 · p. 9

    The study investigates specific goals derived from abstract goals with only one added constraint, not plans involving multiple constraints.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In everyday life, humans often plan their actions by following step-by-step instructions in the form of goal-oriented scripts. Previous work has exploited language models (LMs) to plan for abstract goals of stereotypical activities (e.g., "make a cake"), but leaves more specific goals with multi-facet constraints understudied (e.g., "make a cake for diabetics"). In this paper, we define the task of constrained language planning for the first time. We propose an overgenerate-then-filter approach to improve large language models (LLMs) on this task, and use it to distill a novel constrained language planning dataset, CoScript, which consists of 55,000 scripts. Empirical results demonstrate that our method significantly improves the constrained language planning ability of LLMs, especially on constraint faithfulness. Furthermore, CoScript is demonstrated to be quite effective in endowing smaller LMs with constrained language planning ability.

1 Introduction

Prior script-planning work centers on abstract stereotypical activities, leaving goals with specific constraints understudied. This paper defines constrained language planning, improves LLM planning with over-generate-then-filter, and distills CoScript to train smaller models.

1 Introduction

  • Prior work mainly plans scripts for abstract stereotypical goals, while planning for specific goals with constraints remains understudied.For example, a planner may produce a cake recipe that includes sugar despite a diabetic constraint.Source: §1 · p. 2
  • Constrained language planning requires scripts for specific goals to remain reasonable and faithful to their constraints.The paper illustrates the issue with InstructGPT recommending sugar for a cake intended for diabetic patients.Source: §1 · p. 2
  • Smaller models trained on CoScript can surpass LLM performance on constrained language planning.The dataset contains 55,000 scripts with specific goals and steps.Source: §1 · p. 2
  • 26% increase in accuracy is reported for the over-generate-then-filter method for LLMs.The method selects high-quality scripts from multiple generations.Source: §1 · p. 2
  • The paper uses LLMs to generate CoScript, a dataset of 55,000 scripts, and reports that smaller specialized models achieve performance comparable to LLMs.The authors frame this as distilling a dataset for constrained planning.Source: §1 · p. 2

2 Related Work

Earlier language-planning work and script datasets largely address abstract goals and stereotypical activities. This paper distinguishes its focus on specific goals and uses LLMs as in-context workers to scale reusable planning data.

2 Related Work

  • Prior LM planning studies focus on stereotypical activities or abstract goals, while this work explores specific goals such as making a cake for diabetics.A benchmark on included or excluded objects studied only dozens of cases, without further in-depth analysis.Source: §2 · p. 2
  • Earlier goal-oriented script datasets mainly contain abstract goals with prototypical instructions, rather than specific goals.The paper distinguishes goal-oriented scripts—steps to achieve a goal—from narrative scripts describing event chains.Source: §2 · p. 2
  • Table 1 presents three constraint types and examples that combine each type with an abstract goal to form specific goals.The constraint is highlighted within each example.Source: Table 1 · p. 3
  • LLMs can serve as in-context workers for scaling planning data into a reusable dataset for training smaller models.The paper motivates this as an alternative to costly large-scale crowd-sourcing.Source: §2 · p. 3

3 Definitions

The paper defines specific goals as constrained extensions of abstract goals, then defines constrained language planning as generating scripts faithful to those constraints. These definitions distinguish the task from planning for abstract goals alone.

3 Definitions

  • A goal-oriented script is a list of steps that fulfills a goal, and language planning maps a goal G to a script S.The paper writes this mapping as M : G → S.Source: §3 · p. 3
  • A specific goal inherits an abstract goal and adds information as a constraint that limits its scope.For example, “make a chocolate cake” derives from “make a cake.”Source: §3 · p. 3
  • Constraints are grouped as modifier, method, and intent, and constrained language planning generates a constraint-faithful script for a specific goal.The paper represents the task as S = M(Gc), where Gc is the specific goal.Source: §3 · p. 3

4 Constrained Language Planning with LLMs

The paper uses in-context learning to generate constrained goals and scripts, then filters candidate scripts for constraint faithfulness. Human evaluation finds that this approach improves script quality on specific goals, where baseline performance is often unsatisfactory.

4.1 In-Context Learning for LLMs

  • In-context learning prompts an LLM with a task description, examples, and an input, without requiring gradient updates.wikiHow titles and steps supply examples for the prompting setup.Source: §4.1 · p. 4

4.2 Acquisition of Specific Goals

  • Specific goals are acquired by prompting InstructGPT with examples spanning modifier, method, and intent constraints for each abstract goal.The examples are manually prepared, and constraint types are enumerated to support diversity.Source: §4.2 · p. 4

4.3 Acquisition of Scripts

  • Over-generation samples multiple scripts; a semantic-similarity filter keeps the candidate that ranks the target goal highest, rewarding explicit constraint keywords.The filter compares the target goal with negative goals from the same abstract goal; K = 2 is reported as sufficient in practice.Source: §4.3 · p. 5, §4.4 · p. 5

4.4 Evaluation

  • InstructGPT with our method generates higher-quality scripts by a large margin, while baselines are unsatisfactory on specific goals.InstructGPT outperforms the other baselines, and retrieval from wikiHow does not produce the desired scripts.Source: Table 3 · p. 5, §4.4 · p. 6
  • Human evaluation finds semantic completeness acceptable but constraint faithfulness unreliable for generated scripts; the method improves both dimensions.The evaluation distinguishes missing, repeated, or misordered steps from constraint and coherence errors.Source: §4.4 · p. 6, Figure 3 · p. 6
  • Planning accuracy varies across topic categories, and the method improves accuracy in each category shown.Intent-type constraints have the worst accuracy among the specific-goal constraint types.Source: §4.4 · p. 6, Figure 4 · p. 6

5 Script Distillation from LLMs

Because large language models are costly to deploy and existing datasets do not support constrained planning, the paper distills CoScript from LLM-generated goals and scripts. The resulting dataset contains 55,000 goal-script pairs and exhibits diversity in both scripts and constraint types.

5 Script Distillation from LLMs

  • The dataset is intended to support smaller specialized models because large language models are costly to deploy and manual annotation is demanding.The paper motivates distilling constrained-planning data from LLMs as an alternative to manual dataset annotation.Source: §5 · p. 6

5.1 CoScript: A Dataset for Constrained Language Planning

  • CoScript contains 55,000 specific goals with corresponding scripts, generated from 14,945 article-title seed goals and 34,260 wikiHow examples.The construction pipeline generates specific goals, over-generates scripts, and filters for faithful scripts.Source: §5.1 · p. 7
  • Crowd-reviewed validation and test samples support estimated accuracies of 97.80% for specific goals and 94.98% for constrained script generation.The estimates use annotation data from 5,000 samples.Source: §5.1 · p. 7

5.2 Dataset Analysis

  • CoScript has more specific goals than wikiHow and exceeds human-written proScript in dataset size, script count, steps per script, and unique words.The analysis compares dataset size, unique tokens, goals per abstract goal, and average script steps.Source: §5.2 · p. 7, Table 4 · p. 7
  • CoScript's constraint types are heterogeneous, and hypothetical constraints often begin with “if” or “when” in generated goals.The authors point to this pattern as a potential direction for future counterfactual-reasoning research in language planning.Source: §5.2 · p. 7, Figure 5 · p. 7

6 Constrained Language Planning with Specialized Models

The section evaluates specialized models trained for constrained language planning and compares their performance across training data, retrieval, and constraint types. CoScript-trained models generally outperform wikiHow-trained models, with retrieval and CoScript data each associated with gains in different aspects of performance.

6 Constrained Language Planning with Specialized Models

  • CoScript supports training smaller specialized models for constrained language planning.Source: §6 · p. 7

6.1 Experimental setup

  • The setup compares GPT-2 and T5 trained to generate planning steps, and also tests retrieval-augmented generation.Semantic completeness is measured with BLEU, ROUGE-L, and BERTScore; a binary classifier measures constraint faithfulness.Source: §6.1 · p. 7

6.2 Results

  • Models trained on CoScript generally outperform those trained on wikiHow, while GPT-2 leads on other generation metrics and T5 on faithfulness.The authors suggest CoScript's distillation from InstructGPT may bias its data distribution toward decoder-only models such as GPT.Source: §6.2 · p. 8, Table 5 · p. 8
  • Retrieval augmentation raises T5 (3B)'s manually evaluated correctness from 57% to 70%, while faithfulness falls from 93.00 to 92.53.The authors also report that retrieved examples improve semantic completeness but may undermine constraint faithfulness when models mimic them.Source: §6.2 · p. 8

Faithfulness of Constraints of Different Types

7 Conclusion

The paper concludes that over-generation and filtering improve constrained planning, while CoScript supports smaller models. Its scope and evaluation remain bounded by dataset, model, and assessment limitations.

7 Conclusion

  • CoScript-trained smaller models can even outperform LLMs, while the proposed method improves LLM planning quality for specific goals.The paper presents CoScript as a resource for research on more complex and diverse goals and constraints.Source: §7 · p. 9

Limitations

  • The LLM improvement is post-hoc reranking rather than model fine-tuning, and limited automatic metrics may overestimate or underestimate performance.The authors also do not investigate smaller-model improvements beyond dataset construction and retrieval augmentation.Source: Limitations · p. 9
  • CoScript covers goals with one added constraint, uses InstructGPT-generated data that may favor causal LMs, and retains about 5% uncorrected training errors.The authors manually curate validation and test sets to address distribution bias.Source: Limitations · p. 9

Ethics Statement

  • The study uses U.S.-located Mechanical Turk annotators as a proxy for English competency and pays them above local minimum wage.The stated rate is $6 per hour for 20 samples.Source: Ethics Statement · p. 9
  • CoScript derives from wikiHow under a license permitting noncommercial sharing, copying, distribution, and adaptation.The authors say their academic research use aligns with the license requirements.Source: Ethics Statement · p. 9
  • The dataset spans 19 daily-life categories but excludes medical, legal, and high-stakes financial advice.The exclusions are intended to minimize risks from inaccurate or misleading information.Source: Ethics Statement · p. 10
  • Annotators review scripts for consistency with goals and constraints and revise content to reduce factual errors and misleading information.They are also instructed to discard offensive and harmful data, though prejudicial content may remain unnoticed.Source: Ethics Statement · p. 10

B Implementation Details

The implementation details cover prompt examples for generating specific goals and prompt-format choices for script generation.

B.1 Handpicked Examples for Specific Goal Generation

  • Three examples are reported as a good setting for generating specific goals, with prompts presenting long descriptions as bulleted lists.The experiments compare example counts using accuracy and the total generated specific goals for 100 abstract goals.Source: §B.1 · p. 14, Table 8 · p. 14

B.2 Prompts Format

  • Task prompts improve InstructGPT script generation, and using popular words in prompts further improves their effect.The prompt-format comparison uses 100 samples; Table 9 reports manual-evaluation accuracy.Source: §B.2 · p. 14, Table 9 · p. 14

B.3 Over-generation Hyper-parameters

The over-generation evaluation finds that generating multiple scripts can improve constraint satisfaction, with two generations sufficient in the tested setting.

B.3 Over-generation Hyper-parameters

  • Figure 7’s legends encode the number of constraint-faithful scripts among K over-generated outputs.Human annotators evaluate K generations for each specific goal on 100 samples.Source: Figure 7 · p. 14
  • Over-generation can amplify constraint satisfaction, and K = 2 is sufficient in the reported evaluation.The evaluation randomly samples 100 specific goals; Figure 7 reports the proportion of generated scripts faithful to constraints.Source: §B.3 · p. 14, Figure 7 · p. 14
  • InstructGPT output quality varies substantially, and its scripts may fail to follow constraints.The comparison evaluates scripts from the baseline and the over-generation method.Source: §B.3 · p. 14

B.4 Error Types

The case study contrasts errors in InstructGPT scripts with the over-generation-and-filtering approach; the appendix also describes error categories and generation settings.

B.4 Error Types

  • The evaluation assesses semantic completeness and constraint faithfulness using six defined error types.Table 10 is identified as listing the error types, explanations, and examples.Source: §B.4 · p. 14, Table 10 · p. 15

B.5 Case Study

  • In the case study, InstructGPT scripts sometimes violate constraints or exhibit other errors, while over-generation and filtering increase the likelihood of high-quality outputs.The examples include constraint-faithful scripts that nevertheless have other error types.Source: §B.5 · p. 14

C.1 Generation Hyper-parameters

  • CoScript was generated through the OpenAI API from June 25 to July 5, 2022, at a total cost of about $5,000; when both generations were faithful, one was randomly selected.Source: §C.1 · p. 14
  • CoScript’s category distribution is derived from wikiHow.Source: Figure 8 · p. 15

C.2 Domain Examination

The section examines CoScript for sensitive-domain content, presents qualitative examples of generated goals and scripts, and describes the human-evaluation procedure.

C.2 Domain Examination

  • CoScript, derived from wikiHow, covers 19 daily-life goal categories spanning practical everyday topics while excluding sensitive, high-stakes matters; ChatGPT identified 132 goals (0.24%) as sensitive, which were manually removed and replaced with new specific goals.Source: §C.2 · p. 15

C.3 Qualitative Generations

D Crowd-sourcing Details

  • Crowd workers evaluated whether generated goals retained their abstract goals and constraints, then judged script constraint fit and goal achievement, flagging incorrect steps and error types.Source: D Crowd-sourcing Details · p. 16

Related papers

Loading 2305.05252v5…