PaperlayerView exact revision on arXiv

Source-linked AI summary

How Many Data Samples is an Additional Instruction Worth?

Ravsehaj Singh Puri, Swaroop Mishra, Mihir Parmar, Chitta Baral

arXiv:2203.09161v3cs.CLcs.AIcs.CVcs.LG

TL;DR

Instruction-tuned models still lag supervised task-specific models, while creating datasets or developing models may be difficult for non-experts. This paper augments tasks with alternate instructions and finds gains across learning settings, especially with limited data. On average, an additional instruction is equivalent to about 200 data samples across tasks.

  • Problem

    Instruction-tuned models outperform multitasking baselines but remain far behind supervised task-specific models, while creating datasets and developing models can be difficult for non-experts.

    Source: §1 · p. 1

  • Method

    The authors create multi-variant instructions for 426 tasks and compare learning with variants against single-instruction learning.

    Source: §2.2 · p. 2, §5 · p. 5

  • Results

    17%, 11%, and 11% average gains in task-specific, multi-task, and cross-task settings rise to 26%, 16%, and 11% in low-data regimes, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    An additional instruction is equivalent to about 200 data samples on average across tasks.

    Source: §5 · p. 5

    The analysis covers only English-language tasks, and the authors note that benefits across different language models remain to be tested.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Recently introduced instruction-paradigm empowers non-expert users to leverage NLP resources by defining a new task in natural language. Instruction-tuned models have significantly outperformed multitask learning models (without instruction); however they are far from state-of-the-art task-specific models. Conventional approaches to improve model performance via creating datasets with large number of task instances or architectural changes in the model may not be feasible for non-expert users. However, they can write alternate instructions to represent an instruction task. Is Instruction-augmentation helpful? We augment a subset of tasks in the expanded version of NATURAL INSTRUCTIONS with additional instructions and find that it significantly improves model performance (up to 35%), especially in the low-data regime. Our results indicate that an additional instruction can be equivalent to ~200 data samples on average across tasks.

1 Introduction

Instruction augmentation tests whether users can improve instruction-following performance by writing alternate task instructions rather than relying only on more data or model development. It improves results across learning settings, with larger gains in low-data regimes.

1 Introduction

  • Instruction-tuned models outperform multitasking baselines but remain far behind supervised models trained with task-specific data.Creating high-quality benchmarks, developing models, and training them can be difficult for non-expert users.Source: §1 · p. 1
  • The paper asks whether inexpensive instruction augmentation improves performance, especially where training data are expensive or infeasible to gather.Users can write multiple instructions with different linguistic features, including variants made by replacing words with synonyms.Source: §1 · p. 1
  • Multi-variant instructions can also support robustness evaluation because real users may express the same task instructions in many ways.The paper relates this setup to robustness evaluation using variant data instances.Source: §1 · p. 1
  • The study creates variant instructions for 426 tasks using remaining source-dataset instances beyond the 6,500-instance limit in NATURAL INSTRUCTIONS.It evaluates task-specific, multi-task, and cross-task learning scenarios.Source: §1 · p. 2
  • 17%, 11%, and 11% average gains in task-specific, multi-task, and cross-task settings rise to 26%, 16%, and 11% in low-data regimes, respectively.An additional instruction is equivalent to about 200 data samples on average across tasks.Source: §1 · p. 2

2 Multi-Variant Instruction Dataset

The paper builds a multi-variant instruction dataset on NATURAL INSTRUCTIONS and evaluates how its variants differ from the original tasks. Variants maintain semantic similarity while introducing variation in wording and definition length.

2 Multi-Variant Instruction Dataset

  • The dataset contains multi-variant instructions for 426 NLP tasks built on NATURAL INSTRUCTIONS.The expanded dataset is also known as NATURAL INSTRUCTIONS v2 or SUPER-NATURALINSTRUCTIONS.Source: §2 · p. 2, §2.1 · p. 2

2.1 Variant Instruction Task

  • Variant instruction tasks change the task definition and instances relative to the original NATURAL INSTRUCTIONS task.The schematic marks the entities changed when constructing a variant.Source: §2.1 · p. 2, Figure 1 · p. 2

2.2 Dataset Creation Process

  • Graduate-student participants created variants by preserving definition meaning while changing wording and sampling instances outside the instruction tasks.They could use automated tools or create variants manually.Source: §2.2 · p. 2

2.3 Dataset Properties and Statistics

  • Table 1 summarizes the meta-dataset: variant instruction tasks contain all NATURAL INSTRUCTIONS instances, so the average number of instances per task is higher than 6,500, a stated NATURAL INSTRUCTIONS constraint.Source: §2.3 · p. 2

2.3.1 Semantic Textual Similarity

  • Semantic textual similarity (STS) is computed pairwise between definitions of original instructions and augmented instructions, which represent the same task; Figure 2 reports the mean and standard deviation of these scores.Source: §2.3.1 · p. 2, Figure 2 · p. 3
  • Variants show high semantic similarity for almost all tasks while varying substantially in word choice and definition length.The authors conclude that the variants preserve semantic similarity while providing variation in wording and length.Source: §2.3.1 · p. 3

3 Experimental Setup

Experiments compare single-instruction learning with multi-variant instruction learning using BART-base and T5-base. The evaluation spans task-specific, multi-task, and cross-task settings with varying amounts of training data.

3.1 Models

  • BART-base and T5-base use Huggingface default hyperparameters, with Single Instruction learning as the baseline and Multi-Variant Instruction learning using variants.Both settings use the same number of instances to compare the effect of additional instructions.Source: §3.1 · p. 3

3.2 Experiments

  • Experiments sample 1%, 5%, 10%, 50%, or 100% of task instances and use random 70/20/10 train, test, and development splits.Evaluation uses the test set of original instructions.Source: §3.2 · p. 3
  • Task-specific learning trains and evaluates on the same task, multi-task learning trains on eight tasks, and cross-task learning trains on sampled tasks before evaluation on eight held-out tasks.Cross-task training draws from 274 tasks; multi-task learning uses two positive and two negative examples per task to meet BART-base's token limit.Source: §3.2 · p. 3
  • Evaluation uses Rouge-L, and the SI comparison is limited to one instruction per task because NATURAL INSTRUCTIONS provides exactly one.MVI uses multiple variant instructions.Source: §3.2 · p. 3

4 Results and Analysis

Across task-specific, multi-task, and cross-task settings, multi-variant instructions outperform single-instruction learning, with larger gains in low-data conditions. The analysis estimates that an additional variant instruction is worth about 200 data instances on average and tests equal-data performance and robustness to instruction perturbations.

4.1 Experimental Results

  • MVI outperforms SI by 17% on average in task-specific learning and by 26% in the low-data regime.The low-data average covers fine-tuning with 1%, 5%, and 10% of instances; Figure 3 averages results over three tasks.Source: §4.1 · pp. 3–4, Figure 3 · p. 4
  • MVI outperforms SI by 11% on average in multi-task learning, with gains above 16% in the low-data regime and about 35% at 1% of instances.Figure 4 varies the number of instances.Source: §4.1 · p. 4, Figure 4 · p. 4
  • MVI outperforms SI by 9% on average in cross-task learning when all tasks are included.Figure 5 fixes the number of tasks at 100% while varying instances.Source: §4.1 · p. 4, Figure 5 · p. 4

4.2 Analysis

  • An additional variant instruction is estimated to be worth about 200 data instances on average across task-specific, multi-task, and cross-task settings.The estimate uses MVI performance with 5% of instances as its base; the analysis also reports calculations using 10%.Source: §4.2 · pp. 4–5, Table 3 · p. 5
  • In equal-data analysis, MVI is competitive with SI using N/V instances, where N is the task’s total instances and V its number of instruction variants.The comparison tests whether the estimated value of instruction variants holds when MVI uses fewer instances.Source: §4.2 · p. 5
  • MVI is reported as robust to three test-time instruction perturbations in task-specific learning, with a similar trend in multi-task learning.The perturbations remove the task definition, change the order of positive and negative examples, or remove those examples; SI does not perform equally well on perturbed test sets as on its original test set.Source: §4.2 · p. 5

5 Conclusion

The paper introduces multi-variant instructions for NLP tasks and reports improved performance across task-specific, multi-task, and cross-task learning, with greater effectiveness in the low-data regime. It estimates that one additional instruction is worth about 200 data instances on average.

5 Conclusion

  • Instruction augmentation improves performance in task-specific, multi-task, and cross-task learning and is more effective in the low-data regime.The authors estimate that an additional instruction is equivalent to about 200 instances on average.Source: §5 · p. 5
  • The authors created multi-variant instructions for 426 NLP tasks.Source: §5 · p. 5

Limitations

The paper limits its experiments to BART-base and T5-base and English-language tasks. It also reviews related instruction and prompt-learning work and describes variant examples and dataset-property analyses.

Limitations

  • Experiments use BART-base and T5-base, and the analysis covers English tasks; other models, languages, and augmentation techniques remain for future work.Source: Limitations · p. 5

A Related Work

  • Prior work spans prompt-based task learning and instruction following, while this study focuses on automatic instruction augmentation across data regimes and learning setups.The related-work discussion also notes that prompting has been reported as worth hundreds of data points on average.Source: A Related Work · p. 8

B Example of Variants

C.1 Semantic Textual Similarity

  • The semantic-similarity analysis identifies two exceptions where long original instructions and short variant definitions produce strong variation in similarity scores.The exceptions are task058 and task097.Source: §C.1 · p. 8

C.2 Word-Level Dissimilarity

  • The word-level analysis measures normalized pairwise edit-distance dissimilarity between original and variant definitions and among variants of the same task.The paper reports per-task means and standard deviations in Figure 6.Source: §C.2 · p. 8, Figure 6 · p. 11

C.3 Length Diversity

  • The length-diversity analysis measures the percentage difference between the longest and shortest instruction definitions for each task.The measure is shown in Figure 7.Source: §C.3 · p. 8, Figure 7 · p. 11

D Task-Specific Results

Average results indicate that multi-variant instruction learning improves performance in task-specific learning.

D Task-Specific Results

  • Average results indicate that multi-variant instruction learning improves performance in task-specific learning.The section reports experiments across three named tasks and notes additional generation and classification task categories.Source: D Task-Specific Results · p. 11

E Multi-Task Results

The paper presents its multi-task learning results in Table 9.

E Multi-Task Results

F Cross-Task Results

Cross-task results compare single-instruction learning with multi-variant instruction learning on the original-instruction test set.

F Cross-Task Results

  • The passage identifies Table 12 as reporting cross-task learning experiment results and Figure 9 as comparing single-instruction learning with the authors’ approach in a cross-task setting.Source: F Cross-Task Results · p. 11

G Equal Data Analysis

The equal-data analysis reduces MVI training instances per task according to the number of instruction variants and finds competitive performance in task-specific and multi-task settings.

G Equal Data Analysis

  • MVI uses N/V randomly sampled instances per task, where N is the original task's instance count and V is its number of instruction variants.SI retains its original number of instances.Source: G Equal Data Analysis · p. 11
  • Using only N/V instances, MVI shows competitive performance against single-instruction learning in task-specific and multi-task settings.The comparison uses BART-base; Table 10 summarizes the results.Source: G Equal Data Analysis · p. 11
  • Cross-task experiments compare SI and MVI, with results reported in Table 12 and Figure 9.Figure 9 varies the number of instances and tasks and evaluates on original-instruction test data.Source: G Equal Data Analysis · p. 11

H Robustness Analysis

Multi-variant instruction learning is more robust to instruction changes in the multi-task setting, but individual variants contribute unevenly to performance.

H Robustness Analysis

  • MVI is more robust than SI to paraphrases, spelling mistakes, and grammatical mistakes in multi-task evaluation.The comparison is reflected in smaller performance differences between single-instruction and multi-variant evaluation; Figure 8 presents the robustness comparison.Source: H Robustness Analysis · p. 11, Figure 8 · p. 12

I Contribution of Individual Variants

Related papers

Loading 2203.09161v3…