Source-linked AI summary
The Turking Test: Can Language Models Understand Instructions?
Avia Efrat, Omer Levy
TL;DR
The paper asks whether language models can learn new tasks from natural-language instructions, not only from input-output examples. It introduces the Turking Test to assess instruction following across tasks of varying complexity. GPT-2 performs poorly across all experiments, despite lenient evaluation.
Problem
The paper asks whether language models can learn new tasks from natural-language instructions, which can describe a task more compactly than input-output examples.
Source: §1 · p. 1Method
Results
GPT-2 performed poorly across all experiments, achieving only 2% accuracy on the nth-word task.
Source: §1 · p. 1Takeaways & Limitations
Instructions can explicitly signal what not to do, which examples can only convey implicitly, making instruction understanding a distinct and expressive alternative to few-shot inference.
Source: §1 · p. 2, §6 · p. 7It remains unclear whether traditional language models can capture instruction understanding; they may require grounding beyond linguistic form to follow instructions.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Supervised machine learning provides the learner with a set of input-output examples of the target task. Humans, however, can also learn to perform new tasks from instructions in natural language. Can machines learn to understand instructions as well? We present the Turking Test, which examines a model's ability to follow natural language instructions of varying complexity. These range from simple tasks, like retrieving the nth word of a sentence, to ones that require creativity, such as generating examples for SNLI and SQuAD in place of human intelligence workers ("turkers"). Despite our lenient evaluation methodology, we observe that a large pretrained language model performs poorly across all tasks. Analyzing the model's error patterns reveals that the model tends to ignore explicit instructions and often generates outputs that cannot be construed as an attempt to solve the task. While it is not yet clear whether instruction understanding can be captured by traditional language models, the sheer expressivity of instruction understanding makes it an appealing alternative to the rising few-shot inference paradigm.
1 Introduction
The paper asks whether language models can learn new tasks from natural-language instructions, which can describe tasks more compactly than input-output examples. It introduces the Turking Test and finds that GPT-2 performs poorly across its tasks, even under lenient evaluation.
1 Introduction
- The paper asks whether language models can perform new tasks by conditioning on instructions, as they can on input-output examples.Instructions can describe a task compactly; for example, asking for the fourth word can replace a dozen sentence-and-answer pairs.Source: §1 · p. 1
- The Turking Test evaluates instruction following across tasks ranging from generating dataset examples to listing nouns and retrieving words or characters.The evaluation was lenient, including post-hoc selection of the best generation algorithm and instruction set.Source: §1 · p. 1
- GPT-2 performed poorly across all experiments, achieving only 2% accuracy on the simple task of writing the nth word.The model also tended to ignore explicit restrictions and conditions, and did slightly better on open-ended tasks than on those with well-defined answers.Source: §1 · p. 1
- Instruction understanding is presented as a strict generalization of few-shot inference, with instructions able to convey information examples cannot.Instructions can explicitly signal what not to do, whereas examples can only convey that signal implicitly.Source: §1 · pp. 1–2
- The authors note that it remains unclear whether traditional language models can capture instruction understanding, though doing so could extend the tasks they handle and reduce the need for task-specific datasets.They had not received access to GPT-3 despite submitting an API request in June 2020.Source: §1 · pp. 1–2
2 Instruction Understanding
An instruction understanding task gives a model natural-language instructions that describe a desired output, instantiated with a resource. The paper assesses this setup using task families of varying complexity and refinements to instructions and evaluation intended to improve the model’s chance of success.
2 Instruction Understanding
- An instruction understanding task (IUT) combines a natural-language instruction template with a resource to describe a desired output; the instruction may include input-output examples.A simple example asks for the fourth word of a sentence, while dataset annotation can allow multiple valid outputs.Source: §2 · p. 2
2.1 Experiments and Tasks
- Each experiment contains two or three tasks of a similar nature.The three experiment sets assess different forms of instruction understanding.Source: §2.1 · p. 2
- The experiments cover turking tasks, conditional noun listing, and retrieving a sentence’s nth word or character.Turking tasks use popular datasets such as SQuAD and SNLI; the SQuAD IUT is based on the original SQuAD 1.1 instructions.Source: §2.1 · p. 2, Figure 1 · p. 2
2.2 Authoring the Instructions
- The authors iteratively refined each task’s instructions on a small resource subset, retaining the version on which the model performed best.They applied this process whether or not a task already had instructions.Source: §2.2 · p. 2
2.3 Model
- The experiments use GPT-2 via Hugging Face, with greedy decoding and nucleus sampling at p = 0.9, up to a task-tuned token limit or EOS.At the time of writing, the authors had not been granted access to GPT-3.Source: §2.3 · p. 2
2.4 Evaluation
- Ten college-graduate English-speaking annotators judged whether model outputs followed the instructions.The evaluation relaxed validity rules by accepting valid prefixes and introductory text, and reporting the better of greedy decoding or nucleus sampling with p = 0.9.Source: §2.4 · p. 2
- The oracle generation algorithm was selected separately for each task, so the best choice could differ across tasks.The reported performance used whichever of greedy decoding or nucleus sampling with p = 0.9 led to the better result.Source: §2.4 · pp. 2–3
3 Turking Tasks
The Turking Tasks test whether a model can replace human annotators on three dataset-creation tasks, with lenient output evaluation and analysis of recurring mistakes. Accuracy was very low, and errors included ignoring instructions and producing ungrounded or malformed outputs.
3.1 Tasks
- The tasks replicate SNLI hypothesis writing from photo captions, SQuAD question-answering from Wikipedia paragraphs, and NewsQA question writing from news highlights or inferred articles.They replace human turkers with the model and use development-set resources.Source: §3.1 · p. 3
3.2 Experimental Setting
- 500 outputs per task came from 100 resources with five nucleus generations each, using resources human annotators had found easy.Annotators scored 50 outputs apiece, with 98% observed agreement on a sample spanning all three tasks.Source: §3.2 · p. 4
3.3 Results
- SNLI accuracy was 0.4%, SQuAD accuracy was 0.2%, and NewsQA accuracy was 4.2%, more than an order of magnitude above SQuAD.The authors suggest NewsQA's more open-ended questions may favor language models, while noting that SNLI also includes an instruction example but performs similarly to SQuAD.Source: §3.3 · p. 4
- 94% of SQuAD outputs were not form consistent, and 78% of NewsQA outputs could not be tied to a reasonable news article.Form consistency is a lenient check for whether an output has the right format, even if its content is incorrect.Source: §3.3 · p. 4
3.4 Error Analysis
- The passage reports 24% for NewsQA outputs and 17% for SNLI outputs, versus 3% for SQuAD; it also describes the model continuing the resource instead of following instructions.Source: §3.4 · pp. 4–5
- 19% of SNLI outputs continued the instructions instead of following them.Source: §3.4 · p. 5
- 38% of SQuAD outputs had only generic (23%) or unrelated (15%) questions; the same ungrounded pattern appeared in 9% of NewsQA outputs.Source: §3.4 · p. 5
- Instruction examples grounded 11% of SNLI and 6% of NewsQA outputs, always inaccurately; 49% of NewsQA outputs still contained a forbidden “why” or “how” question.Source: §3.4 · p. 5
4 Listing Nouns
After GPT-2 performed poorly on turking tasks, the authors tested noun listing with and without simple conditions. Accuracy remained low, and adding conditions further reduced performance.
4.1 Tasks
- The tasks require listing all nouns in a sentence, either unconditionally or when they satisfy a conjunction of two properties drawn from case, initial sound, and grammatical number; 12 conditions are possible.Source: §4.1 · p. 6
4.2 Experimental Setting
- Pairing each resource with all 12 possible conditions and filtering cases with no qualifying nouns yielded 100 unconditioned and 689 conditioned examples; p = 0.9 nucleus generation led to 0% accuracy on both tasks, so only greedy results are reported.Each input received one greedy and one nucleus generation, with outputs limited to the resource sentence’s token count plus 20.Source: §4.2 · p. 6
4.3 Results
- 9% accuracy on unconditioned noun listing fell to 3.5% when a condition was added; successes favored shorter sentences, with no significant differences across conditions.Table 2 reports noun-listing performance, and Figure 6 compares sentence lengths for all resources and those correctly handled.Source: Table 2 · p. 6, §4.3 · p. 6, Figure 6 · p. 6
4.4 Error Analysis
- Repeated copies of the resource or instruction appeared in 10% of unconditioned and 30% of conditioned outputs, always producing invalid answers.Nucleus sampling also produced 0% accuracy on both tasks.Source: §4.4 · pp. 6–7
5 Retrieving Elements by Index
After noun listing still required linguistic knowledge, the authors tested whether GPT-2 could follow instructions to retrieve a word or character by index.
5.1 Tasks
- The tasks requested the nth word or nth character of a sentence, with the character instruction also tested using the word “letter.”Retrieving an indexed element was presented as simple enough to implement in one line of Python.Source: §5.1 · p. 7, §5 · p. 7
5.2 Experimental Setting
- For each resource sentence, the study limited word indices to 20 and character indices to 40, excluded whitespace-character indices, and generated 1,664 outputs per task using greedy and p = 0.9 nucleus decoding.Only greedy results were reported because nucleus sampling was significantly worse on both tasks.Source: §5.2 · p. 7
5.3 Results
- The model achieved at most 2% accuracy on either retrieval task, below the baseline of writing the most common word or character.The tasks retrieve the nth word or character; for the character task, a slight wording change to “nth letter” yielded better results during instruction optimization.Source: §5.3 · p. 7, §5.1 · p. 7
5.4 Error Analysis
- Repetitions occurred in 46% of nth-word and 59% of nth-character outputs, always making them incorrect; sampled repetition patterns also differed between the nearly identical tasks.Of sampled nth-character repetitions, 96% began with the resource; none of the sampled nth-word repetitions did.Source: §5.4 · p. 7
6 Discussion: The Instruction Paradigm
The instruction paradigm strictly generalizes few-shot inference because instructions can include examples and convey information examples cannot. The paper presents following instructions as a serious challenge for language models, with grounding beyond linguistic form as one possible requirement.
6 Discussion: The Instruction Paradigm
- Instructions strictly generalize few-shot inference because they can include examples and communicate information examples alone cannot.The paper's SNLI and NewsQA tasks embed expected input-output examples in their instructions.Source: §6 · p. 7
- Instructions can explicitly rule out invalid outputs, while examples can only convey that negative signal implicitly.The NewsQA instruction, for example, prohibits “why” questions.Source: §6 · p. 8
- GPT-2 failed a series of increasingly simple instructions; larger or instruction-trained models may do better, while grounding beyond linguistic form may be needed.The paper says it remains unclear whether traditional language models can capture instruction understanding.Source: §6 · p. 8
A All Instructions
The appendix collects the full instructions used across Sections 3–5, including the resource substitutions used to instantiate each task.
A All Instructions
- The appendix contains the full instructions from Sections 3–5, each additionally appended with a newline.Source: A All Instructions · p. 9
- The turking-task instructions use SNLI captions, SQuAD Wikipedia paragraphs, or NewsQA article highlights as resources.Source: Figure 7 · p. 10, Figure 8 · p. 10, Figure 9 · p. 11
- The noun-listing instructions cover unconditioned and conditioned tasks, with the latter instantiated for 12 possible conditions.Source: Figure 11 · p. 12, Figure 10
- The retrieval instructions instantiate nth-word indices from 1 to 20 and nth-character indices from 1 to 40, limited by sentence length.Source: Figure 12 · p. 12, Figure 13 · p. 12