PaperlayerView exact revision on arXiv

Source-linked AI summary

CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs

Taha Aksu, Devamanyu Hazarika, Shikib Mehri, Seokhwan Kim, Dilek Hakkani-Tür, Yang Liu, Mahdi Namazifar

arXiv:2311.17376v1cs.CL

TL;DR

Dialog models may struggle with prompts that combine multiple constraints, and large-scale compositional demonstrations are difficult to author manually. CESAR modularizes and automatically combines dialog-task prompt components, and InstructDial++ provides a benchmark with basic and composite tasks. Experiments report improvements on compositional and atomic tasks when composite-task data is included in training.

  • Problem

    Models may perform well on individual dialog constraints yet struggle to satisfy them together, while the number of compositions grows exponentially and requires manual instruction work.

    Source: §1 · pp. 1–2

  • Method

    CESAR modularizes dialog prompts and automatically combines task components, and InstructDial++ applies it to 63 datasets with 86 basic and 68 composite tasks.

    Source: §1 · p. 2

  • Results

    Composite-task training improves performance on seen and unseen compositions and on atomic tasks under similar data budgets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments show that including composite data in training enhances both compositional performance and atomic-task performance.

    Source: §1 · p. 2, §7 · p. 9

    InstructDial++ is not comprehensive, and the work limits its dataset scope to dialog-specific data rather than multitasking with non-dialog data.

    Source: §8 · p. 10

Abstract

from arXiv · show

Instruction-based multitasking has played a critical role in the success of large language models (LLMs) in multi-turn dialog applications. While publicly available LLMs have shown promising performance, when exposed to complex instructions with multiple constraints, they lag against state-of-the-art models like ChatGPT. In this work, we hypothesize that the availability of large-scale complex demonstrations is crucial in bridging this gap. Focusing on dialog applications, we propose a novel framework, CESAR, that unifies a large number of dialog tasks in the same format and allows programmatic induction of complex instructions without any manual effort. We apply CESAR on InstructDial, a benchmark for instruction-based dialog tasks. We further enhance InstructDial with new datasets and tasks and utilize CESAR to induce complex tasks with compositional instructions. This results in a new benchmark called InstructDial++, which includes 63 datasets with 86 basic tasks and 68 composite tasks. Through rigorous experiments, we demonstrate the scalability of CESAR in providing rich instructions. Models trained on InstructDial++ can follow compositional prompts, such as prompts that ask for multiple stylistic constraints.

1 Introduction

Dialog models may handle individual constraints yet struggle when prompts combine them, while scaling valid compositions requires substantial manual effort. CESAR automatically combines prompt components, and experiments report gains on both compositional and atomic tasks from composite-task training.

1 Introduction

  • A model that performs well on individual control dimensions may still struggle to satisfy them simultaneously when their composition was absent from training.The example combines keyword requirements with an ask-question dialog act.Source: §1 · p. 1
  • The number of task compositions grows exponentially with the number of atomic tasks, making large-scale demonstrations costly to author manually.The passage identifies adding appropriate instructions for each new composition as a source of human labor.Source: §1 · p. 2
  • Naively combining task instructions and controls can yield infeasible, invalid, or overly idiosyncratic cross-task compositions.The paper calls for reliable composition without manual effort.Source: §1 · p. 2
  • CESAR modularizes dialog prompts into inputs, constraints, and outputs, then automatically combines components to form compositional prompts.Figure 1 illustrates combining keyword-controlled generation with act-grounded generation.Source: Figure 1 · p. 1, §1 · p. 2
  • InstructDial++ expands the benchmark to 63 datasets, with 86 basic tasks and 68 composite tasks.It updates InstructDial with additional atomic tasks and datasets and uses CESAR to introduce composite tasks.Source: §1 · p. 2
  • Composite-task training improves performance on seen and unseen compositions and on atomic tasks under similar data budgets.The paper reports these benefits for compositional tasks in fine-tuning.Source: §1 · p. 2

2 Related Work

Prior work unified tasks, structured compositional examples, and controlled dialog outputs, but the cited approaches did not explore task composition through unification or directly target its training-data scalability. The section also contrasts ChatGPT's and InstructDial's performance as constraint counts increase.

Instruction Tuning

  • Instruction-tuning work scaled tasks or targeted downstream dialog, but InstructDial lacked curated compositional tasks (Wei et al., 2021; Mishra et al., 2022; Chung et al., 2022a; Gupta et al., 2022).The related-work discussion positions CESAR as enabling large-scale training of compositional tasks.Source: §2 · p. 2

Unified Grounding

  • UnifiedSKG unifies 21 SKG tasks into a text-to-text format, and ConvLab3 proposes a shared format for task-oriented dialog datasets; neither explores task composition from unification efforts.UnifiedSKG demonstrates improved performance through multitask learning.Source: §2 · pp. 2–3

Closed- vs. Open-access Models in Complex Dialog Tasks

  • ChatGPT handles two constraints but struggles at three or more, whereas InstructDial cannot satisfy scenarios with two or more constraints.The comparison concerns instruction following for dialog response generation.Source: Figure 2 · p. 3

Compositional Generalization via Structure

  • Structured approaches use constituent parts and decision templates, logical forms, or task-specific prompts to support compositional generalization (Chen et al., 2022b).The examples span compositional fine-tuning, semantic parsing, and cross-task commonalities.Source: §2 · p. 3

Dialog Control

  • Dialog-control work combines task prompts, compositional codebooks, separately embedded attributes, or staged decoding, while CESAR targets scalable compositional training data.The passage says these approaches do not directly compare to CESAR.Source: §2 · p. 3

3 Motivation

The section motivates compositional dialog training and reports a preliminary comparison of models trained with and without compositional tasks. The compositional model performs better across training sizes, with a reported positive effect on atomic-task performance as well.

Closed- vs. Open-access Models in Complex Dialog Tasks

  • The motivation contrasts satisfactory simple-composite-task results from ChatGPT with DIAL-T0’s difficulty, motivating additional resources for open-access models.Closed- versus open-access status is defined by whether model parameters and data are publicly available.Source: §3 · p. 3

Can Compositional Demonstrations Improve Performance?

  • The preliminary experiment manually combines four response-control tasks: beginning phrase, ending phrase, length, and keyword.The two training sets are equal in size, and both models use the same number of training steps; the Compositional model mixes atomic and compositional tasks.Source: §3 · pp. 3–4
  • The Compositional model outperforms the Baseline across training sizes, and including compositional tasks also positively affects atomic-task performance.Figure 3 compares compositional accuracy over varying training sizes; the atomic-task comparison is referenced as Figure 8.Source: §3 · p. 4, Figure 3 · p. 4

4 CESAR

CESAR represents dialog tasks through structured components, then composes task prompts by combining atomic tasks. Its task definitions distinguish atomic from compositional tasks and permit increasingly complex compositions when relevant atomic tasks are available.

4 CESAR

  • CESAR treats dialog items as information units and maps them to context, state, evidence, action, or response components.Examples of dialog items include utterances, speaker states, intents, stylistic attributes, summaries, revisions, and external knowledge snippets.Source: §4 · p. 4

4.1 CESAR Framework

  • A CESAR task is a text-to-text task whose input and output linearize dialog components in a specified structure, separated by a dash.Inputs include an instruction, dialog context, and a multiset of dialog components; the output is one nonempty component from S, E, A, or R.Source: §4.1 · p. 4
  • This work limits CESAR tasks to one output component; generating multiple outputs, such as a summary and response, is deferred.The broader multi-output setup is left for future work.Source: §4.1 · p. 4, §4.2 · p. 5

4.2 CESAR Tasks

  • An n-D task has n dialog items; atomic tasks are 0-D or 1-D, while compositional tasks have at least two dialog items.A 0-D task still includes an instruction and dialog context, which may be empty.Source: §4.2 · p. 5
  • CESAR’s composition operation combines two i-D tasks into an (i + 1)-D task, enabling arbitrarily complex tasks when relevant atomic tasks are available.The framework also formulates chain-of-thought variants by shifting dialog items from input to output, but leaves their experimentation for future work.Source: §4.2 · p. 5

5 InstructDial++

InstructDial++ expands the dialogue benchmark with new datasets and atomic tasks, then uses CESAR’s task structure to map existing tasks and generate viable compositions. Its general instructions also support programmatic task prompts, including discriminative tasks.

5.1 New Tasks and Datasets

  • The benchmark adds 15 datasets and 42 atomic tasks; after these additions, it contains 86 tasks across 65 datasets.Most newly introduced atomic tasks derive from CESAR’s structure, which permits multiple tasks to be defined on the same dataset.Source: §5.1 · p. 6

5.2 Mapping InstructDial++ to CESAR

  • CESAR instructions identify the dialog components to use and the component to generate, allowing instructions for compositional tasks to be generated programmatically.This differs from InstructDial’s use of unique instructions for each task.Source: §5.2 · p. 6
  • CESAR can specify discriminative tasks by placing candidate labels in the state component, as in emotion classification.The framework’s tasks are generative in nature, but the format also accommodates this discriminative setup.Source: §5.2 · p. 6
  • CESAR organizes downstream tasks as 0-D and 1-D task types, with InstructDial++ covering all four 0-D types and 10 of the 1-D types.The types group tasks by their output objective and, for 1-D tasks, an additional grounding component.Source: §5.2 · pp. 6–7
  • CESAR selects viable 2-D task compositions using predefined rules, yielding 68 downstream tasks across seven 2-D CESAR task types.The rules are intended to prevent infeasible combinations; the chord-wheel edges represent compositions between atomic tasks.Source: Figure 5 · p. 6, §5.2 · p. 7
  • Prompts randomize the positions of context and grounding items, while keeping the instruction first and the target section last.The target section is denoted ψ; the randomized sections are C and Λ.Source: §5.2 · p. 7

6 Experiments

Experiments compare CESAR-FLAN-xxl with public and ChatGPT models on atomic and compositional dialogue tasks. CESAR outperforms baselines on every tested composition, while the robustness experiment finds advantages over the Naive Composer on all seen compositions and most unseen tasks and compositions.

6.1 Setup

  • The experiments use ChatGPT, four public or InstructDial-based models, and CESAR-FLAN-xxl trained on InstructDial++.Test instances are the same across models, but prompts follow each model’s training format.Source: §6.1 · p. 7, §6.3 · p. 8

6.2 Tasks and Metrics

  • Evaluation covers eight atomic tasks and nine binary compositions, using task-specific metrics and multiple metrics when combined accuracy is unclear.For persona-based plus ends-with generation, the reported measures are ends-with accuracy and Rouge-L.Source: §6.2 · p. 8

6.3 Results

  • CESAR structure outperforms the Naive Composer on all seen compositions and most unseen tasks and compositions.The models train on the same four atomic tasks and three compositional tasks; the Naive Composer’s task combinations are manually selected.Source: §6.3 · p. 8, Table 6 · p. 9
  • CESAR’s atomic-task performance is comparable to, and often better than, the baselines, without a claimed discrete advantage.Table 4 presents the atomic-task results.Source: §6.3 · p. 8, Table 4 · p. 9
  • CESAR-FLAN-xxl outperforms the baselines on every tested task composition.Table 5 reports the compositional-task evaluation results.Source: §6.3 · p. 8, Table 5 · p. 9
  • Because the Naive Composer’s tasks were manually selected, its comparison with CESAR tests robustness to CESAR structure rather than the framework’s ability to detect viable compositions.The authors explicitly caution that the experiment is not a direct comparison of the two approaches.Source: §6.3 · p. 9

7 Conclusion

CESAR modularizes dialogue tasks to generate complex composite instructions, and InstructDial++ supplies a benchmark containing atomic and composite tasks. Experiments report improvements on compositional prompts, while the authors find only minimal risk of test contamination.

7 Conclusion

  • CESAR modularizes downstream dialogue tasks under one format, enabling programmatic generation of complex composite instructions.The authors position this as addressing a compositional capability gap between public models and ChatGPT.Source: §7 · p. 9
  • The authors estimate minimal test-set contamination, with at most 100 test instances seen by the model in the worst case.Their estimate follows from limited overlap between the DailyDialog versions used in FLAN and the paper’s evaluation.Source: §7 · p. 9
  • InstructDial++ contains 63 datasets, 86 atomic task definitions, and 68 composite task definitions.It builds on InstructDial with new tasks and datasets and uses CESAR to populate composite tasks.Source: §7 · p. 9
  • Experiments report improved handling of complex prompts, including gains on compositional and atomic tasks and better results on most unseen tasks and compositions.The conclusion also reports that CESAR structure outperforms baselines in a majority of compositions.Source: §7 · p. 9

8 Limitations

The authors identify limits in dataset coverage, experimental scope, and the available control signals and action fields.

8 Limitations

  • InstructDial++ is not comprehensive; additional datasets such as Contrack and PRESTO could broaden its coverage.The authors also note that the benchmark was expanded only as far as they could manage.Source: §8 · p. 10
  • The experiments are limited to dialog-specific data and do not explore negative conditions, whose evaluation is challenging.Source: §8 · p. 10
  • The work uses existing dataset grounding features, while its action fields are mostly lexical, limiting the controls explored.The authors suggest a more comprehensive set of actions would be better.Source: §8 · p. 10

A Training Details

Training uses sampled atomic and compositional examples, fixed sequence limits, and a shared two-epoch optimization setup for DIAL-FLAN-xxl and CESAR-FLAN-xxl.

A Training Details

  • InstructDial and InstructDial++ training sampled 5,000 instances per atomic task; InstructDial++ also sampled 1,000 instances per CESAR-generated compositional task per dataset.Source: A Training Details · p. 16
  • Input and output sequences were capped at 1,024 and 128 tokens, respectively, with longer sequences truncated and shorter ones padded.Source: A Training Details · p. 16
  • Both models trained for two epochs on eight A100 GPUs, using batch size 10 per device, accumulation steps of 4, learning rate 5e-05, and AdamW.Source: A Training Details · p. 16

B Compositional Capabilities of Closed Source Models

Closed-source models show varying compositional performance, while CESAR’s composition rules specify how compatible dialog tasks are combined into composite tasks.

B Compositional Capabilities of Closed Source Models

  • GPT-4 is described as the most reliable of the three closed-source models, with Claude v2 comparable and GPT-3.5 somewhat behind.All three show compositional capability, but each also has inaccuracies or deviations from expected outputs.Source: B Compositional Capabilities of Closed… · p. 16

C Composition Rules

  • CESAR’s ten current composition rules constrain which structured dialog tasks can be combined automatically.Source: C Composition Rules · p. 16
  • One rule combines two ICA-R tasks into an ICAA-R task whose response output encapsulates both actions, such as beginning-with and keyword-controlled generation.Source: C Composition Rules · p. 16
  • The rule’s common fields preserve shared modules such as dialog context and response, while its target field specifies the resulting output.For ICA-R compositions, the shared response field becomes the ICAA-R output; the rule applies across variants that satisfy its constraints.Source: C Composition Rules · p. 16

D Benhmarking ChatGPT on Atomic and Compositional Tasks

The supplementary ChatGPT benchmarking uses naturally formatted prompts and reports comparisons, while the CESAR extension describes how reasoning components can be incorporated into tasks.

D Benhmarking ChatGPT on Atomic and Compositional Tasks

  • ChatGPT benchmarking results are supplementary because the model is closed-access and its architecture, parameters, and API behavior limit reproducibility.Source: D Benhmarking ChatGPT on Atomic and… · p. 16
  • ChatGPT was evaluated with naturally formatted prompts matching DIAL-FLAN, with one-shot prompts adding an in-context example.Source: D Benhmarking ChatGPT on Atomic and… · p. 16
  • ChatGPT results are lower than CESAR and DIAL-FLAN-xxl in the reported atomic and compositional evaluations, but the authors caution that trained models may exploit dataset and preprocessing traits.For example, punctuation spacing in beginswith_generation can cause an exact-match evaluation to reject ChatGPT’s response; endswith_generation may also penalize continued generation after the required phrase.Source: D Benhmarking ChatGPT on Atomic and… · pp. 16, 18
  • On three tasks with a reduced test set, GPT-4 performed notably better than GPT-3.5, and adding contextual examples considerably boosted performance.Table 10 is identified as the evaluation of those tasks with more in-context samples and the GPT-4 API; the supplied passages give no metric values.Source: D Benhmarking ChatGPT on Atomic and… · p. 18, Table 10 · p. 18

E Chain of Thought Potential in CESAR

F ChatGPT Evaluation Prompts

This section describes ChatGPT-based evaluation prompts and inventories CESAR downstream tasks, covering emotion-grounded responses, atomic grounding tasks, and compositional 2D tasks.

F ChatGPT Evaluation Prompts

  • Emotion-grounded generation was evaluated with ChatGPT by classifying generated emotions for accuracy and assigning response-quality scores to detect emotion naming without emotional expression.The quality score addresses cases where a response names an emotion directly rather than conveying it through the response.Source: F ChatGPT Evaluation Prompts · p. 19
  • In-context examples set the emotion-classification and response-quality prompts, with templates shown in Tables 12 and 13.Source: F ChatGPT Evaluation Prompts · p. 19, Table 13 · p. 19, Table 12
  • Although CESAR accommodates chain-of-thought, the experiments excluded it and used an empty Λ′ set.Source: F ChatGPT Evaluation Prompts · p. 19

G Can Naive Conjunctions provide Complex Instructions?

  • The Naive Composer appends single-task instructions to form complex prompts; the experiments manually selected which tasks to combine and compared these prompts with CESAR prompts.Source: G Can Naive Conjunctions provide… · p. 20
  • Naive conjunctions do not scale reliably because blindly combining tasks with the same dialog context can create nonsensical, rare, or infeasible tasks.In the example, dialog-act prediction feeds into dialog-act-grounded generation, making the second task's answer part of its own input.Source: G Can Naive Conjunctions provide… · p. 20

H Structured vs. Natural Prompts

Loading 2311.17376v1…