PaperlayerView exact revision on arXiv

Source-linked AI summary

Chain-of-Instructions: Compositional Instruction Tuning on Large Language Models

Shirley Anugrah Hayati, Taehee Jung, Tristan Bodding-Long, Sudipta Kar, Abhinav Sethy, Joo-Kyung Kim, Dongyeop Kang

arXiv:2402.11532v3cs.CL

TL;DR

Existing instruction datasets mostly focus on single instructions, while complex instructions can contain multiple sequential subtasks. The paper introduces CoI and tunes models on chained instructions, reporting better performance on multi-instruction tasks and generalization to unseen and longer chains.

  • Problem

    Most instruction datasets contain single instructions, while LLMs’ ability to handle complex compositional instructions is less explored.

    Source: Introduction · p. 1

  • Method

    The authors define CoI as sequential subtasks whose outputs feed into the next and build a dataset and tuning approach for these chains.

    Source: Introduction · pp. 1–2

  • Results

    CoI-tuned models outperform single-instruction and sequential-compositional baselines on multi-instruction tasks, with reported generalization to unseen longer chains and downstream tasks.

    Source: Introduction · p. 2, Conclusion and Future Work · p. 7

  • Takeaways & Limitations

    CoI-tuning offers a way to train models for compositional instructions by having them produce incremental outputs across subtasks.

    Source: Introduction · p. 2

    Models show poor performance on Spanish summaries, which the authors associate possibly with a lack of Spanish as the first training subtask.

    Source: Generalization to Downstream Composite… · p. 7

Abstract

from arXiv · show

Fine-tuning large language models (LLMs) with a collection of large and diverse instructions has improved the model's generalization to different tasks, even for unseen tasks. However, most existing instruction datasets include only single instructions, and they struggle to follow complex instructions composed of multiple subtasks. In this work, we propose a novel concept of compositional instructions called chain-of-instructions (CoI), where the output of one instruction becomes an input for the next like a chain. Unlike the conventional practice of solving single instruction tasks, our proposed method encourages a model to solve each subtask step by step until the final answer is reached. CoI-tuning (i.e., fine-tuning with CoI instructions) improves the model's ability to handle instructions composed of multiple subtasks as well as unseen composite tasks such as multilingual summarization. Overall, our study find that simple CoI tuning of existing instruction data can provide consistent generalization to solve more complex, unseen, and longer chains of instructions.

Introduction

Chain-of-Instructions (CoI) frames complex instructions as sequential subtasks, where each output becomes the next input. The paper introduces a way to construct and tune on these chains, reporting gains on compositional, unseen, and longer-chain tasks.

Introduction

  • CoI defines compositional instructions as sequential subtasks whose outputs feed into the next subtask, while some task pairs cannot compose because their formats mismatch.For example, a summary can feed title generation, but not a Data-to-Text task.Source: Introduction · p. 1
  • The paper creates a CoI dataset and an automatic construction framework using existing single-instruction data, then evaluates models on single and compositional tasks.The framework uses in-context learning, and the method encourages incremental outputs at each step of a chain.Source: Introduction · p. 2
  • CoI-tuned models outperform single-instruction and sequential-compositional baselines, with gains extending to unseen longer chains and downstream tasks.The paper also reports that models fine-tuned only on single instructions never outperform CoI-tuned models on multi-instruction tasks.Source: Introduction · p. 2

Chain-of-Instructions

CoI formalizes chains as composed instructions and provides a pipeline for creating chained training examples from single-task instructions. The dataset spans multiple chain lengths, while the authors inspect composability quality and task patterns.

Formulation

  • A chain of length k composes k instructions so that each next instruction consumes the preceding subtask’s output.The formulation writes this as Ii+1 ◦ Ii(Xi) = Yi+1 for each step.Source: Formulation · p. 2

Automatic Dataset Creation Pipeline

  • The authors use GPT 3.5 Turbo for the instruction-composition procedure and state that the procedure can be reproduced with other strong LLMs.The model is selected for reasonable price and adequate quality at the time of the study.Source: Automatic Dataset Creation Pipeline · p. 3
  • An LLM shortens SUP-NATINS instruction descriptions from an average of 62.19 words to 14.33 words.The authors motivate shortening because the source instructions are lengthy and detailed.Source: Automatic Dataset Creation Pipeline · p. 3
  • The composability check first applies heuristic validity filters, then tests whether an LLM can generate the next instruction’s output from the preceding input.For example, classification tasks can only be the final subtask.Source: Automatic Dataset Creation Pipeline · p. 3
  • For valid pairs, the generated output of the second instruction becomes its ground truth; CoI3 chains two CoI2s, and the same approach builds longer chains.The authors define chains from two and three instructions as CoI2 and CoI3, respectively.Source: Automatic Dataset Creation Pipeline · p. 3
  • Manual inspection found 75% of sampled CoI2 compositions and 59% of sampled CoI3 compositions valid.The authors inspected 100 instances for each chain length.Source: Automatic Dataset Creation Pipeline · p. 3

CoI Dataset

  • CoI datasets include chain lengths 2–5, with lengths 4 and 5 reserved for testing; generation tasks can occur in either position, while format-specific and closed-ended tasks tend to be second.The dataset has 970 unique category pairs at length 2 and 418 unique category triplets at length 3.Source: CoI Dataset · p. 3

Experiment Setup

The experiments compare CoI-tuned Alpaca-7B and Mistral-7B-Instruct with untuned, prompted, and single-instruction baselines across compositional, difficult single-task, and multilingual summarization tests.

Models and baselines

  • Alpaca-7B and Mistral-7B-Instruct are the fine-tuned base models, compared with untuned models, CoT and least-to-most prompted models, and CoI1 models.Evaluation reports ROUGE-L, LLM-as-judge preferences, and blind human pairwise comparisons.Source: Experiment Setup · p. 3

Data creation

  • Figure 3 depicts CoI2 data creation, using an LLM for instruction summarization and composability checking, with an example dataset instance at right.Output 1 in Step 2 comes from the original SUPNATINST data.Source: Figure 3 · p. 4

Instruction analysis

  • Figure 4 is a t-SNE visualization of sentence embeddings for the most frequent CoI2 compositional instructions.The accompanying analysis reports that generation tasks can occur in either chain position, while format-specific and close-ended tasks mostly occur second.Source: Figure 4 · p. 4

Test sets

  • The compositional test suites cover chain lengths σ = 2, 3, 4, and 5, alongside BIG-Bench Hard for difficult single instructions.The downstream evaluation uses English-to-French and Spanish-to-English WikiLingua summarization, language pairs unseen in CoI training.Source: Experiment Setup · p. 4

Results

CoI-tuned models generally outperform baselines on compositional tests, with the best training configuration varying by chain length. Human judgments and results on BIG-Bench Hard also favor CoI-tuning, while incorrect intermediate training outputs degrade performance.

Performance on In-domain Composite Tasks (CoI2,3)

  • On CoI2-test, Alpaca and Mistral CoI12 perform best; on CoI3-test, CoI123 variants perform best, and compositional tuning generally outperforms baselines.CoI3-tuned Alpaca is an exception, slightly underperforming CoI1-tuned Alpaca on CoI2-test.Source: Table 4 · p. 5, Performance on In-domain Composite… · p. 5

Results

  • Training with incorrect second- or third-subtask outputs degrades model performance, indicating that correct intermediate outputs matter during training.Source: Results · p. 5

Performance on In-domain Composite Tasks (CoI2,3)

  • LLM preferences over CoI1 are 69.90% for Alpaca CoI12 and 70.92% for Mistral CoI12 on CoI2-test, and 81.04% and 60.00% for CoI123 on CoI3-test.The values are respectively for Alpaca and Mistral on each test set.Source: Performance on In-domain Composite… · p. 5
  • Human evaluators prefer CoI-tuned outputs across all CoI2 and CoI3 test sets for both Alpaca and Mistral.The evaluation sampled 200 instances per test set and model and used eight NLP-knowledgeable computer science students.Source: Figure 5 · p. 5, Performance on In-domain Composite… · p. 5
  • CoI models generally outperform baselines across intermediate tasks, with the strongest results on the first subtask, followed by the second and third.On CoI3, second-subtask performance exceeds CoI's second-subtask performance on CoI2; the authors conjecture that easier preceding subtasks or later-subtask context may help.Source: Performance on In-domain Composite… · p. 5

Generalization to Unseen Single Tasks

Generalization to Longer Chains (CoI4,5)

CoI tuning with shorter chains helps models handle unseen longer chains, and CoI12 improves multilingual summarization results against baselines, with outcomes varying by language and model.

Generalization to Longer Chains (CoI4,5)

  • Training on chains of length 2 or 3 helps models understand unseen length-4 and length-5 chains, with less performance drop than CoI1 or non-fine-tuned baselines.The authors posit that training on shorter compositional chains supports understanding of longer composed tasks.Source: Generalization to Longer Chains (CoI4,5) · p. 6

Generalization to Downstream Composite Tasks

  • For English-to-French summarization, CoI12 generates more valid target outputs than the baselines and obtains higher ROUGE for source summaries.Table 6 reports multilingual summarization results on 300 instances.Source: Generalization to Downstream Composite… · p. 6, Table 6 · p. 7
  • For Spanish-to-English summarization, CoI12 Mistral outperforms the baseline on all ROUGE-L scores, while Alpaca does not improve source- or target-summary ROUGE-L.Source: Generalization to Downstream Composite… · p. 7
  • CoI performs better on English-to-French than Spanish-to-English summarization, which the authors relate to English-origin translation tasks in training and possibly limited Spanish-first-subtask examples.They conjecture that adding more Spanish tasks during fine-tuning could address the Spanish-summary weakness.Source: Generalization to Downstream Composite… · p. 7

Related Work

Prior instruction-tuning datasets primarily address single instructions, while compositional NLP work studies sequential subproblems in other settings. CoI differs by chaining instructions so each subtask's output becomes the next one's input.

Instruction tuning

  • Prior instruction datasets include human-written and LLM-assisted examples, but address single-instruction problems (Wang et al. 2022; Wang et al. 2023).Source: Related Work · p. 7
  • Unlike earlier LLM-generated instruction data, CoI uses an LLM to judge whether instructions compose and generate responses for subsequent subtasks (Schick and Schütze 2021; Peng et al. 2023; Shao et al. 2023).Source: Related Work · p. 7

Compositional problems in NLP

Conclusion and Future Work

The paper introduces Chain-of-Instructions and a dataset for the task, with an automatic construction pipeline and evaluations on generated and downstream tasks. It highlights the multi-instruction results and proposes instruction decomposition and broader task composition as future directions.

Conclusion and Future Work

CoI Dataset Statistics

The CoI dataset combines many task categories into numerous pairs and triplets, with longer chains using fewer eligible categories to ensure composability.

CoI Dataset Statistics

  • CoI2 contains 41 first-subtask categories, 67 second-subtask categories, and 970 unique category pairs.The dataset reports an average of 34.25 words per CoI2 instruction.Source: CoI Dataset Statistics · pp. 9–10
  • CoI3 contains 39 first-subtask, 35 second-subtask, and 61 third-subtask categories, with 418 first-to-second pairs, 700 second-to-third pairs, and 2,148 unique triplets.The average CoI3 instruction contains 51.00 words.Source: CoI Dataset Statistics · p. 10
  • CoI3 has fewer first- and second-subtask categories to ensure composability across the longer chain; dataset construction selects at most three instances per category pair.Source: CoI Dataset Statistics · p. 10

Prompting Details

This section introduces prompting details for the approach.

Summarizing and generalizing a single instruction

The prompt asks an LLM to generalize and shorten a detailed instruction, with an example for question generation.

Checking compositionality of two instructions and obtaining the second output

The appendix describes CoI data construction and evaluation variants, including results for concise instructions and for chains with incorrect later outputs. It also provides prompts, examples, and intermediate-result tables used in the study.

Checking compositionality of two instructions and obtaining the second output

LLM as a Preference Judge

  • The appendix supplies prompts and examples for preference judging, fine-tuning, BBH tasks, and English–French and Spanish–English summarization.The preference judge compares generated outputs with a ground-truth output; fine-tuning uses Alpaca-7B and Mistral-7B-Instruct, and the multilingual examples show chained summarization and translation.Source: LLM as a Preference Judge · p. 11, Fine-tuning Details · p. 11, WikiLingua Examples · p. 12

Data Generation Details

  • CoI data use SUP-NATINS tasks connected through English; the authors sampled task pairs, checked composability, and expanded the data into two- and three-instruction training and test sets.They report 1,341 selected tasks, 3,024 candidate pairs, and 1,115 composable pairs; the expanded sets include 2,933 training and 588 test instances for two-instruction problems, and 2,187 training and 480 test instances for three-instruction problems.Source: Data Generation Details · pp. 12, 14

Downstream Task

  • For baseline multilingual outputs without explicit subtask boundaries, language identification is used to distinguish source-language from target-language summaries.Source: Downstream Task · p. 14

Full results

  • Concise-CoI is created by prompting an LLM to summarize a compositional instruction in at most 20 words while preserving its meaning.Its training instance uses the same input and output as CoI; the appendix gives an example of this shortening process.Source: Creating concise-CoI · p. 11, Full results · p. 14
  • Incorrect second- or third-task outputs matter for compositional test performance, although they have little effect on single-task problems.The irrelevant-CoI variant assigns a random original-dataset output to the second task instead of the correct output; Table 9 covers compositional tests and BBH.Source: Full results · p. 14, Table 9 · p. 16
  • C-CoI123 Mistral performs better than CoI models on every reported dataset except BBH; on BBH, C-CoI12 and C-CoI123 Alpaca outperform their CoI counterparts.The section presents these comparisons in Table 4 and also identifies shorter instructions as potentially helpful for unseen single tasks.Source: Full results · p. 14

Full Intermediate Results

Loading 2402.11532v3…