PaperlayerView exact revision on arXiv

Source-linked AI summary

Goal-Oriented Script Construction

Qing Lyu, Li Zhang, Chris Callison-Burch

arXiv:2107.13189v3cs.CL

TL;DR

Evaluating whether models understand complete scripts remains an open question. The paper introduces goal-oriented script construction, studies generation and retrieval on multilingual wikiHow data, and reports that state-of-the-art models can produce complete scripts in and out of domain, with room for improvement.

  • Problem

    How to evaluate script understanding remains open, and it is unclear whether current models can generate complete scripts.

    Source: §1 · p. 1

  • Method

    The paper proposes GOSC, asks models to construct ordered steps from a goal, and evaluates generation and retrieval approaches using multilingual wikiHow scripts.

    Source: §1 · pp. 1–2

  • Results

    State-of-the-art models can produce complete scripts in and out of domain, though the results leave substantial room for improvement.

    Source: §9 · p. 9

  • Takeaways & Limitations

    The wikiHow-trained approach can be applied to other datasets and distant domains, including military and political domains.

    Source: §1 · p. 2

    The retrieval pipeline has long run-time with large candidate pools because it requires pairwise goal-step inference.

    Source: §8 · p. 9

Abstract

from arXiv · show

The knowledge of scripts, common chains of events in stereotypical scenarios, is a valuable asset for task-oriented natural language understanding systems. We propose the Goal-Oriented Script Construction task, where a model produces a sequence of steps to accomplish a given goal. We pilot our task on the first multilingual script learning dataset supporting 18 languages collected from wikiHow, a website containing half a million how-to articles. For baselines, we consider both a generation-based approach using a language model and a retrieval-based approach by first retrieving the relevant steps from a large candidate pool and then ordering them. We show that our task is practical, feasible but challenging for state-of-the-art Transformer models, and that our methods can be readily deployed for various other datasets and domains with decent zero-shot performance.

1 Introduction

The paper frames complete scripts as useful knowledge for task-oriented systems, while noting that script understanding remains difficult to evaluate. It introduces GOSC, a multilingual dataset, and generation and retrieval baselines to study script construction.

1 Introduction

  • Script knowledge is proposed as useful for task-oriented dialogue agents, but how to evaluate it remains an open question.The introduction also asks whether current models can generate complete scripts.Source: §1 · p. 1
  • GOSC asks a model to construct a complete sequence of steps from a goal, targeting holistic script understanding rather than isolated script knowledge.The authors argue that understanding existing scripts should support producing new ones from absorbed knowledge.Source: §1 · p. 1
  • The paper introduces a multilingual wikiHow script corpus, extending prior English-focused script learning to 18 languages.The corpus spans how-to articles across a wide range of domains.Source: §1 · p. 1
  • The authors compare generation from scratch with retrieval and ordering of relevant steps, finding GOSC challenging but feasible for state-of-the-art Transformers.They also report strong zero-shot performance on three classic script corpora and deployment to distant domains.Source: §1 · p. 2

2 Related Work

Prior work studies narrative scripts and goal-directed procedural scripts, but its evaluation tasks often measure only fragments of script understanding. The paper positions GOSC as a more holistic task and highlights multilingual learning and cross-domain transfer.

2 Related Work

  • Narrative-script work distills descriptive event knowledge from texts, while procedural scripts describe events that occur to achieve a goal.Prior procedural-script datasets are generally small-scale and often based on crowdsourced event-sequence descriptions.Source: §2 · p. 2
  • Existing evaluation remains fragmented: tasks test script completion, event clustering or ordering, alignment, and next-event prediction rather than full understanding.Recent generation tasks have mainly focused on specialized domains such as stories or recipes.Source: §2 · p. 2
  • Related goal-step studies mostly use separate binary-classification or multiple-choice tasks with few negative candidates, whereas GOSC uses open-ended retrieval or generation.The paper notes that earlier work did not include extrinsic evaluation on other datasets or domains.Source: §2 · p. 3
  • The paper distinguishes its work through complete-script construction, multilingual learning, dataset scale and quality, and transfer to other domains without training.Its corpus is described as higher quality than mined narrative scripts and lower cost and larger scale than crowdsourced event-sequence descriptions.Source: §2 · p. 3

3 Goal Oriented Script Construction

GOSC takes a goal and an expected script length and returns an ordered list of steps. It has generation and retrieval settings, distinguished by whether candidate steps are supplied.

3 Goal Oriented Script Construction

  • Given a goal and expected number of steps, GOSC constructs a complete script as an ordered list of steps against a ground-truth reference.The expected length provides a hint about the desired granularity.Source: §3 · p. 3
  • In the Generation setting, the model creates the entire script from scratch.In Retrieval, it selects and orders a subset from a supplied candidate set.Source: §3 · p. 3

4 Multilingual WikiHow Corpus

The authors extend an English wikiHow corpus with articles in 17 additional languages to form a multilingual resource. They extract goal-oriented scripts from article structure, project ordering labels across languages, and define a held-out test split.

4 Multilingual WikiHow Corpus

  • The existing English wikiHow corpus is extended by crawling articles in 17 other languages, yielding a multilingual corpus.The listed languages include Spanish, Portuguese, Chinese, German, French, Russian, Italian, Indonesian, Dutch, Arabic, Vietnamese, Thai, Japanese, Korean, Czech, Hindi, and Turkish.Source: §4 · p. 3
  • Each script uses the article title without “How to” as its goal, method or part headers as sections, and step-paragraph headlines as steps.The top-level wikiHow category is also recorded, and the example is drawn from an English article about eating at a sit-down restaurant.Source: §4 · p. 3, Figure 2 · p. 3
  • English ordering labels are projected to linked articles in other languages; unmatched articles default to unordered, and ordering is required only for ordered articles.The article hierarchy is flattened, treating steps from all sections as steps in the script.Source: §4 · p. 3
  • Experiments hold out 10% of articles in each language for testing and use the remaining 90% for training and development.For retrieval evaluation, candidates are drawn from steps in the test set.Source: §4 · p. 3

5 Models

The paper evaluates two Transformer-based systems for constructing scripts from goals: generation produces steps directly, while retrieval selects and orders steps from candidate pools. The retrieval system separates step relevance prediction from pairwise ordering and also tests multitask and cross-lingual variants.

5.1 Generation Approach: Multilingual T5

  • mT5 is finetuned to generate a goal-conditioned sequence of all script steps, combining relevant-step inference and ordering in one generation process.At inference, it generates a list of steps from a test-set goal.Source: §5.1 · p. 4

5.2 Retrieval Approach: Step-Inference-Ordering Pipeline

  • The Step-Inference-Ordering pipeline first scores candidate steps for relevance to a goal, then orders retrieved steps through pairwise comparisons.The inference model retains the top l steps by confidence; the ordering model sorts by how often each step is ranked before others.Source: §5.2 · p. 4
  • Multitask learning shares an encoder between inference and ordering, while cross-lingual zero-shot transfer evaluates English-trained models on non-English data.The systems use separate task classifiers in the multitask variant.Source: §5.2 · pp. 4–5

6 In-Domain Evaluation

The in-domain evaluation combines automatic metrics and human judgments on the multilingual wikiHow dataset. Retrieval supplies more correct steps in human evaluation, while generation orders steps better; automatic results also show language variation and room for improvement.

6.2 Auto Evaluation for Retrieval Setting

  • Around 30% accuracy and 0.2 Kendall’s τ characterize the best retrieval system, while multitask and cross-lingual zero-shot variants improve ordering, including +0.17 τ for Korean MTL and Thai C0.Results are generally similar across languages, with substantial room for improvement.Source: Table 2 · p. 5, §6.2 · p. 6

6.1 Auto Evaluation for Generation Setting

  • ρ = −0.856 (p = 1e−5) links perplexity with article count, and ρ = −0.669 (p = 2e−4) links it with mC4 token count; BERTScore F1 is .82 for English and .65–.72 elsewhere.The authors note that BERTScore may miss step alignment and ordering, motivating human evaluation.Source: §6.1 · pp. 5–6

6.3 Human Evaluation

  • Six annotators edited 30 outputs per assigned language by deleting or reordering steps; XLM-R Step Ordering converged for only three languages across tested settings.The edited scripts were assessed for correctness, completeness, and orderliness.Source: §6.3 · p. 6
  • Retrieval scripts contain more correct steps, but their ordering is significantly worse than generation scripts in human evaluation.The comparison uses the same 30 gold scripts across six languages.Source: Table 3 · p. 6, §6.3 · p. 6

6.4 Qualitative Examples

  • 50% step accuracy in “Make a Quotebook” still includes a plausible near-match, while generation omits essential steps; the Santa example shows hallucinated and repetitive details.The authors report that these patterns recur in a larger sample.Source: §6.4 · pp. 6–7, Figure 5 · p. 7

7 Zero-shot Transfer Learning

The retrieval pipeline trained on wikiHow is tested zero-shot on four English script corpora and in a KAIROS case study. The reported results show non-trivial transfer across datasets and a substantial domain shift.

7.1 Quantitative Evaluation

  • 30%–60% of steps are accurately retrieved and around 40% correctly ordered across four target corpora, close to or better than in-domain English results.The random baseline averages 0.013 Accuracy and 0.004 τ.Source: Table 4 · p. 7, §7.1 · p. 7
  • The transfer evaluation selects each scenario’s longest event sequence description, then applies an English wikiHow-trained XLM-RoBERTa pipeline directly to four target corpora.The corpora cover human activities and military or political scenarios.Source: §7.1 · p. 7, §7 · p. 7

7.2 Case Study: The Bombing Attack Scripts

  • For five IED goals, the case study extracts KAIROS ontology events from retrieved documents, converts them to text with templates, and retains steps above a confidence threshold.OneIE extracts event instances; the default retention threshold is 0.95.Source: §7.2 · p. 8
  • 0.735 average Correctness and 0.404 Orderliness were reported for five IED scripts despite the wikiHow-to-KAIROS domain shift.The retrieved steps were sensible, though some were misordered, including a ManufactureAssemble event that should precede the others.Source: §7.2 · p. 8, Figure 6 · p. 8

8 Limitations

The paper identifies limitations in representing events as natural-language steps and in its modeling and evaluation approaches. These include semantic duplicates, computational costs, and imperfect automatic and human evaluation.

8 Limitations

  • Semantically similar natural-language steps can both be retrieved, even when returning both is undesirable.The paper notes this difficulty for steps with nearly identical meanings that are both valid for a goal.Source: §8 · p. 8
  • The Step-Inference-Ordering pipeline has long runtimes on large candidate pools because it requires pairwise goal-step inference.The authors suggest filtering irrelevant steps with similarity heuristics as an alternative, and identify paraphrasing as a way to eliminate semantic duplicates.Source: §8 · p. 9
  • Automatic retrieval metrics miss inexact matches, while generation metrics may not capture step ordering and one annotator per language risks subjective judgments.The paper calls for better evaluation methods, noting that crowdsourcing is costly and hard to standardize.Source: §8 · p. 9

9 Conclusion and Future Work

The conclusion reports that current state-of-the-art models can produce complete scripts both in- and out-of-domain, but with substantial room for improvement. The paper also describes its multilingual corpus and evaluation procedures for script construction and component modules.

9 Conclusion and Future Work

  • State-of-the-art models produce complete scripts in- and out-of-domain, with substantial room for improvement.The authors propose future work on script usefulness and granularity, and on utility for downstream tasks requiring automated reasoning.Source: §9 · p. 9

A Corpus Statistics

  • The corpus statistics table reports language-level counts, orders languages by article count, and treats all sections of each article as one script.The supplied passages describe the table’s scope but give no specific cell values.Source: A Corpus Statistics · p. 11, Table 5 · p. 12

B Evaluation Details

  • The task takes a goal and expected script length as input; retrieval-setting outputs are assessed for step content and ordering.The task defines the target as an ordered list of steps with a ground-truth reference.Source: B Evaluation Details · p. 12
  • Content accuracy counts predicted steps found in the ground truth, while Kendall’s τ measures ordering among overlapping predicted and gold steps.The ordering measure uses concordant and discordant pairs across the ordered-list intersection.Source: B Evaluation Details · p. 12
  • Retrieval is evaluated with recall and normalized discounted cumulative gain at position k, while ordering is tested on gold steps using Kendall’s τ.The retrieval module ranks candidate steps by relevance; the ordering module receives the ground-truth steps directly.Source: B Evaluation Details · p. 12
  • Generation-setting evaluation uses test-set perplexity, based on sequence log-likelihood, and averages metrics across scripts.The passages also define a predicted step’s true relevance as 1 when it occurs in the ground-truth steps and 0 otherwise.Source: B Evaluation Details · p. 12

C Modeling Details

The modeling setup fine-tunes pretrained models for GOSC and selects models using validation performance. Experiments use a shared development split and report different selection metrics for retrieval and generation.

C Modeling Details

  • All experiments hold out 5% of training data for development and run on an NVIDIA GeForce RTX 2080 Ti using FP16 with O1 optimization.Worst-case runtimes across all languages are 3 hours to 5 days for retrieval and 2 hours to 1 day for generation.Source: C Modeling Details · p. 13
  • mBERT, XLM-RoBERTa, RoBERTa, and mT5-Large are the pretrained models used, with sequence classification for the first three and official finetuning scripts for mT5.The listed checkpoints have 168M, 270M, 125M, and 1B parameters, respectively.Source: C Modeling Details · p. 13
  • Models are selected by highest validation performance after grid search; retrieval uses contracted-script accuracy, while generation uses perplexity.The search varies learning rate from 1e−5 to 5e−8, batch size from 16 to 128 when possible, and epochs from 3 to 10.Source: C Modeling Details · p. 13
  • Table 6 reports multitask mBERT retrieval results, with bold marking results higher than single-task mBERT.Source: Table 6 · p. 13

D Additional Results

The appendix points to tables with additional in-domain retrieval results and presents more constructed-script examples. The examples compare retrieval and generation outputs, while the tables cover model, module, and transfer settings.

D Additional Results

E More Qualitative Examples

  • The paper presents two additional scripts comparing mBERT retrieval-setting outputs with mT5 generation-setting outputs side by side; Figures 7–8 show English examples, and Figures 9–10 show Chinese examples.Source: E More Qualitative Examples · p. 13
  • The supplementary materials include 20 in-domain examples for each language and all five out-of-domain examples for the Bombing Attack scenario.Source: E More Qualitative Examples · p. 13

Related papers

Loading 2107.13189v3…