PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructEval: Systematic Evaluation of Instruction Selection Methods

Anirudh Ajith, Chris Pan, Mengzhou Xia, Ameet Deshpande, Karthik Narasimhan

arXiv:2307.00259v2cs.CLcs.AI

TL;DR

Prompt details affect ICL performance, making instruction choice an important prompt-engineering question. InstructEval evaluates instruction-selection methods across models and tasks, finding that simpler instructions often outperform automatic induction. The authors call for more systematic evaluation and release the suite to support it.

  • Problem

    Because prompt details affect ICL performance, choosing instructions is an important prompt-engineering question.

    Source: §1 · p. 1

  • Method

    InstructEval evaluates seven instruction-selection methods across 13 open-source LLMs, four families, nine tasks, and three accuracy plus two sensitivity metrics.

    Source: §1 · p. 2

  • Results

    Task-agnostic and expert-written instructions often outperform automatic induction; task-agnostic methods perform strongly in few-shot settings, while expert instructions appear crucial for good zero-shot accuracy.

    Source: §5.4 · p. 8

  • Takeaways & Limitations

    The authors call for more systematic and consistent evaluation of instruction-selection methods and release InstructEval to facilitate it.

    Source: §5.4 · p. 8, §6 · p. 8

    Automatic methods may lack generalizability across models and tasks or require substantial hyperparameter tuning to work well broadly.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

In-context learning (ICL) performs tasks by prompting a large language model (LLM) using an instruction and a small set of annotated examples called demonstrations. Recent work has shown that precise details of the inputs used in the ICL prompt significantly impact performance, which has incentivized instruction selection algorithms. The effect of instruction-choice however is severely underexplored, with existing analyses restricted to shallow subsets of models and tasks, limiting the generalizability of their insights. We develop InstructEval, an ICL evaluation suite to conduct a thorough assessment of these techniques. The suite includes 13 open-sourced LLMs of varying scales from four model families, and covers nine tasks across three categories. Using the suite, we evaluate the relative performance of seven popular instruction selection methods over five metrics relevant to ICL. Our experiments reveal that using curated manually-written instructions or simple instructions without any task-specific descriptions often elicits superior ICL performance overall than that of automatic instruction-induction methods, pointing to a lack of generalizability among the latter. We release our evaluation suite for benchmarking instruction selection approaches and enabling more generalizable methods in this space.

1 Introduction

Although prompt details are critical to ICL, instruction choice remains underexplored, and prior evaluations make methods difficult to compare across models, tasks, and metrics. InstructEval provides a broad evaluation suite and finds that simpler instruction choices often outperform automatic induction.

1 Introduction

  • Instruction choice is underexplored despite its significance, while ICL prompts typically combine a task-specific instruction, annotated demonstrations, and a test example.Prompt construction details remain important to ICL performance.Source: §1 · pp. 1–2
  • Prior evaluations use inconsistent protocols and narrow model and task selections, often emphasizing classification and zero-shot accuracy over other task types and ICL metrics.Some studies cover one model or a single model family, making relative performance difficult to compare.Source: §1 · p. 2
  • InstructEval evaluates seven instruction-selection methods on 13 open-source autoregressive LLMs from four families and nine tasks, using three accuracy and two sensitivity metrics.The evaluated methods range from trivial baselines and manually curated instructions to automatic methods.Source: §1 · p. 2
  • Curated manual and task-agnostic instructions achieve better aggregated performance than automatic induction, while including instructions in few-shot prompts usually hurts ICL at the model scales considered.The authors also report that omitting instructions may be optimal in few-shot settings and curated manual instructions in zero-shot settings.Source: §1 · p. 2
  • The authors release the suite to support systematic study of prompt-engineering questions their work does not explicitly address.The release is intended to aid further study beyond the questions covered in this work.Source: §1 · p. 2

2 Related Work

Prior work evaluates language models or instruction-selection methods with limited overlap in models and tasks. InstructEval aims to enable broader comparisons, including across accuracy and sensitivity metrics.

2 Related Work

  • BigBench and HELM evaluate language models rather than systematically evaluating prompting methods; this work assesses a broader range of instruction-selection approaches than probability-based selection alone.The comparison includes trivial baselines, curated instructions, and automated methods.Source: §2 · p. 2

Automated Prompt Engineering Methods

  • Automated prompt-engineering studies span perplexity heuristics, LLM-induced instructions, and reinforcement-learning prompts, but their small, weakly overlapping evaluations hinder objective comparison.InstructEval aims to compare popular instruction-selection methods across diverse tasks and models and includes sensitivity metrics; other prompting facets are left to future work.Source: Automated Prompt Engineering Methods · pp. 2–3
  • ICL research finds that demonstration selection and ordering affect few-shot accuracy, while sensitivity to demonstration permutations can reduce reliability, particularly in low-resource domains.These studies motivate attention to prompt sensitivity beyond accuracy alone.Source: Automated Prompt Engineering Methods · pp. 2–3
  • The authors distinguish instruction engineering from other prompting facets, which they defer to future work.Their evaluation focuses on instruction-selection approaches and accuracy and sensitivity metrics.Source: Automated Prompt Engineering Methods · p. 3

3 Evaluation Suite

InstructEval evaluates instruction-selection methods across diverse tasks and models using accuracy and sensitivity metrics. Its aggregation procedures compare accuracy relative to instruction-population means and average sensitivity scores across models.

3.1 Prompt format

  • Prompts consist of an instruction, annotated demonstrations, and an unsolved test example, with flexible support for demonstrations, templates, and custom strings.Source: §3.1 · p. 3

3.2 Metrics

  • The evaluation measures zero-shot, few-shot, and perturbation accuracy, plus sensitivity to demonstration selection and ordering across 16 random choices.Perturbation accuracy tests random capitalization, spacing, contractions, and common misspellings in test inputs.Source: §3.2 · p. 3

3.3 Aggregating metrics across Models

  • Mean relative gain aggregates accuracy across models, while sensitivity scores are averaged directly because their values vary little across methods.Source: §3.3 · p. 4
  • Accuracy varies more with model and task choice than with instruction choice.Source: §3.3.1 · p. 4

3.3.1 Accuracy metrics

  • For accuracy, relative gain measures how far a score exceeds the instruction population’s mean; task-level values average these gains across models.Source: §3.3.1 · p. 4

3.3.2 Sensitivity metrics

  • Sensitivity aggregation averages each method’s raw sensitivity scores across models; the authors avoid relative-gain aggregation for these secondary metrics.Source: §3.3.2 · p. 4

3.4 Tasks

  • The nine tasks span classification, multiple-choice question-answering, and generative question-answering; tasks are challenging but above random-chance performance.Summarization is excluded because it is challenging to assess objectively.Source: §3.4 · p. 5

3.5 Models

  • The suite uses 13 autoregressive LLMs from four model families, spanning 1.1 billion to 20 billion parameters.Source: §3.5 · p. 5

4 Experimental setup

The setup compares task-agnostic, manually written, and automatically synthesized instruction-selection approaches.

Task-agnostic instructions

  • Task-agnostic baselines omit instructions or use generic prompts such as “Complete the following task:”.The null condition contains demonstrations and a test example in few-shot settings, but only an unanswered test example in zero-shot settings.Source: §4 · p. 5

Manual task-specific instructions

  • Manual task-specific instructions include PromptSource templates and ad hoc instructions paraphrased from PromptSource seeds.PromptSource is a public collection of manually curated templates for 170+ datasets; the ad hoc condition uses ChatGPT-generated paraphrases and random sampling.Source: §4 · p. 5

Automatically synthesized task-specific instructions

  • Automatic methods include SPELL low-perplexity selection, APE validation-based refinement, and RLPrompt reinforcement-learning-based induction.SPELL selects each model’s least-perplexity instructions from ChatGPT paraphrases; APE refines generated prompts using held-out accuracy. RLPrompt is assessed on MCQ but not GQA tasks because it is not directly applicable to generation.Source: §4 · pp. 5–6

5 Results

Instruction-selection performance varies across models, tasks, and settings: simple or manually curated instructions often outperform automatic methods, while instruction effects are generally larger in zero-shot settings.

5.1 Less sophisticated instruction selection methods tend to show higher accuracy

  • Null and Generic instructions achieve the highest aggregated few-shot accuracy in 5/9 tasks and perturbation accuracy in 6/9, while PromptSource leads zero-shot accuracy in 6/9 tasks.Source: §5.1 · p. 6
  • Automatic task-specific methods lead zero-shot performance on no task and lead few-shot performance in only 2/9 tasks, inconsistently; RLPrompt is above average in 5/7 tasks in both settings but is usually outperformed by simpler methods.Source: §5.1 · p. 6

5.2 Ranges of variation of aggregated scores

  • Zero-shot instruction choice shows larger ranges of accuracy gains and losses than few-shot choice, consistent with instructions conveying task semantics and demonstrations supplying more of that understanding in few-shot settings.Source: §5.2 · p. 6
  • Accuracy variation by instruction choice is greatest for AG News and Emotion in few-shot settings, and for GQA tasks in zero-shot settings.Source: §5.2 · p. 6
  • Selectional and permutational sensitivities vary substantially by task but are roughly consistent across methods within a task; most methods have lower permutational than selectional sensitivity.Source: §5.2 · p. 6, Table 5 · p. 7

5.3 Analysis

  • Across model scales, task-agnostic and PromptSource trends persist, while large models show smaller few-shot accuracy-gain ranges than small models for every task type.Source: Table 6 · p. 8, §5.3 · p. 8

5.4 Discussion

  • At the evaluated scales, task-agnostic and expert-written instructions often outperform automatically synthesized ones; the latter may lack broad generalizability or require substantial tuning.Source: §5.4 · p. 8
  • Method rankings vary by model and task, so practitioners may need model- and task-specific experimentation; the authors call for more comprehensive, consistent evaluations.Source: §5.4 · p. 8

6 Conclusion

The conclusion presents InstructEval as a broad assessment of instruction-selection generalizability and releases it to support more systematic evaluation.

6 Conclusion

  • The paper describes its work as the broadest attempt, to its knowledge, to systematically study popular ICL instruction-selection and induction methods’ generalizability.Source: §6 · p. 8
  • Across diverse models and tasks, task-agnostic instructions, expert-written instructions, and omitting instructions tend to perform well more consistently.Source: §6 · p. 8
  • InstructEval is released with coverage of 13 autoregressive LLMs and nine tasks spanning classification, multiple-choice QA, and generative QA to facilitate systematic evaluation.Source: §6 · p. 8

A Variation across model families

Across model families, leading instruction-selection alternatives vary by family and task type, even though broad trends for task-agnostic and PromptSource instructions persist.

A Variation across model families

  • Task-agnostic and PromptSource instructions tend to dominate few-shot and zero-shot settings across model families, but the best-performing alternative often changes with model family and task type.Source: A Variation across model families · p. 12
  • APE and RLPrompt are above average for some model-family and task-type combinations, but this performance does not consistently extend to others.Source: A Variation across model families · p. 12
  • The authors interpret the inconsistent results across families and task types as indicating limited generalizability for automatic instruction-induction methods.Source: A Variation across model families · p. 12

B.1 Evaluation

The evaluation controls variability with repeated random seeds, randomly sampled few-shot demonstrations, and fixed task-specific prompt templates.

B.1 Evaluation

  • Each method is rerun with five independent random seeds for every task and, where applicable, model, with results averaged across the five instructions.Source: §B.1 · p. 12
  • Few-shot experiments use K = 6 demonstrations randomly sampled from each task’s training set.Source: §B.1 · p. 12
  • Experiments use fixed task-specific prompt templates that place the tested instruction before annotated demonstrations and a test example.Source: §B.1 · p. 12

B.2 Instruction selection methods

The evaluation compares instruction-selection approaches ranging from sampled or induced instructions to prompts with no task-specific instruction. It also documents the suite’s task templates and example instructions.

B.2 Instruction selection methods

  • PromptSource uses randomly sampled repository instructions, while Ad hoc samples from 40 ChatGPT paraphrases and Low Perplexity selects each model’s lowest-perplexity instruction.Low Perplexity reranks ChatGPT paraphrases of PromptSource instructions using SPELL.Source: §B.2 · p. 12
  • APE uses DaVinci for induction and the simpler search strategy; RLPrompt uses default hyperparameters and extends instruction induction to MCQ tasks as cloze completions.The simpler APE version is used because the authors report only marginal accuracy gains from Monte-Carlo search.Source: §B.2 · pp. 12–13
  • Table 7 reports mean relative gains for zero-shot and few-shot accuracy by model family and task type, with a count of each method’s highest-performance tasks.Source: Table 7 · p. 13
  • Table 8 lists the evaluated tasks and their demonstration templates.Source: Table 8 · p. 13
  • Task-agnostic evaluation omits instructions from the prompt.Source: §B.2 · p. 13
  • Table 9 provides example instructions produced by each method for AG News.Source: §B.2 · p. 13

C Drawbacks of aggregation techniques used in previous work

Prior aggregation approaches have trade-offs: head-to-head win rates preserve rankings but hide metric magnitudes, while cross-task normalization requires thresholds that are infeasible to obtain for this evaluation.

C Drawbacks of aggregation techniques used in previous work

Related papers

Loading 2307.00259v2…