PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Task Descriptions

Orion Weller, Nicholas Lourie, Matt Gardner, Matthew E. Peters

arXiv:2011.08115v1cs.CL

TL;DR

The paper addresses the gap between systems that need many task-specific examples and humans who can perform new tasks from instructions. It proposes learning from natural-language task descriptions, instantiates the framework in the ZEST dataset, and evaluates models at the task level across unseen tasks. The best T5 model scores 12% on ZEST, while the authors identify few-shot extensions as future work.

  • Problem

    Current supervised NLP systems generally learn from large sets of task-specific examples, requiring new training data for new tasks and potentially exploiting biases in training distributions.

  • Method

    The paper synthesizes prior zero-shot work, proposes learning from task descriptions, and introduces ZEST, which uses question-form descriptions and task-level evaluation across multiple inputs and task types.

  • Results

    12%: the best T5-based model’s score on ZEST, compared with a 42% human performance estimate.

  • Takeaways & Limitations

    Zero-shot learning from complex task descriptions remains a significant challenge for current NLP systems, leaving substantial room for improvement.

  • Takeaways & Limitations

    The paper focuses on zero-shot learning, while few-shot settings with task descriptions and examples remain an avenue for future work.

Abstract

from arXiv · show

Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task. Moreover, the dataset's structure tests specific types of systematic generalization. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers.

1 Introduction

The paper proposes learning unseen NLP tasks from natural-language descriptions rather than task-specific examples. It introduces the ZEST dataset and finds that current systems perform far below humans on this setting.

  • Motivation: Learning from task descriptions is proposed as a general-purpose NLP paradigm that enables zero-shot solving of unseen tasks without collecting a new training set for each task.The authors argue this could improve accessibility for practitioners and reduce biases associated with memorizing training-distribution artifacts.
  • Contributions: The paper synthesizes prior zero-shot NLP work, formalizes the prediction problem, and extends evaluation to the more complex task of slot filling.Earlier work covered text classification, entity typing, and relation extraction.
  • Dataset: ZEST formulates task descriptions as questions paired with text passages, evaluates the resulting task function across many inputs, and tests systematic generalization across selected task-description axes.This structure is intended to assess whether a model solves the task rather than only a single question–passage instance.
  • Results: 12%: the best T5-based model’s ZEST score, compared with a 42% human performance estimate.The authors characterize zero-shot learning from complex task descriptions as a significant challenge for current NLP systems.

2 Learning from task descriptions

The framework reframes NLP from fitting task-specific examples to understanding natural-language task descriptions and applying them to unseen tasks. It formalizes task competence, motivates task-level evaluation, and distinguishes whole-task generalization from prior zero-shot and multitask settings.

  • Learning from task descriptions: Learning from examples requires a new dataset for each task, discards the task definition after annotation, and usually keeps the task fixed at test time.The framework instead exposes the task description to the model alongside the input and targets zero-shot generalization to unseen tasks.
  • Learning from task descriptions: The framework shifts the objective from fitting a probability distribution to understanding a task description well enough to apply it to arbitrary inputs.This reframing treats task descriptions as executable specifications for desired outputs.
  • Task competence: Task competence evaluates each task using its complete set of input-output pairs, then averages whether the model reaches each task’s competence threshold.When all thresholds equal T, the resulting measure is denoted C@T.
  • Discussion: Unlike conventional zero-shot learning, which generalizes to unseen classes within the same task, this framework generalizes to entire unseen tasks described by natural language.Multitask learning similarly handles multiple tasks but learns each from examples and does not generalize to unseen tasks.

3 Instantiating the Framework

The paper defines a framework for evaluating whether NLP systems can perform unseen tasks from task descriptions, varying inputs, outputs, tasks, and descriptions to test systematic generalization. ZEST supports mixed output types and controlled tests including paraphrase, semantic flips, composition, and unseen output structures.

  • Output structure: ZEST mixes discrete classes, typed or untyped spans, span relationships, and NA outputs, with the task description specifying the allowable output set.NA marks inputs for which the described task cannot be solved, an important condition when arbitrary tasks and documents are combined.
  • The framework evaluates generalization across inputs, outputs, tasks, and task descriptions rather than only performance on familiar task-example distributions.It is intended to make progress toward systems that can perform unseen NLP tasks from descriptions without additional task-specific training.
  • Four controlled tests probe paraphrase, semantic flips, composition, and output structure beyond unseen tasks on unseen inputs.These variants isolate whether models preserve task meaning across wording changes, distinguish minimally altered descriptions, compose tasks, and produce unseen structured outputs.
  • Paraphrase: Paraphrase evaluation keeps the underlying task fixed while changing its description, testing whether the same input receives the same output despite altered wording.The design also increases lexical distance between retrieved inputs and task descriptions to discourage simple lexical matching.
  • Semantic flips: Semantic-flip evaluation changes a task description minimally while changing its meaning, testing whether systems distinguish descriptions that retain most of their wording.The output set remains largely unchanged, making the altered task meaning the key distinction.

4 Collecting ZEST

ZEST was built as a challenge dataset through crowdsourced task collection, document retrieval, and answer annotation. Its question-form tasks are varied across controlled generalization settings and paired with roughly twenty documents each to measure task-level competence.

  • ZEST was designed as a challenge dataset to demonstrate where current systems fail at learning tasks from descriptions.The collection involved task-question creation, relevant-document retrieval, and answer annotation for task-document pairs.
  • Each base task is varied across four generalization areas, and approximately 20 documents per task enable calculation of a competency metric.The dataset therefore evaluates performance across multiple inputs for the same task rather than on a single question-document instance.
  • Tasks are written as questions because question annotation is established for crowdsourcing and may support transfer from existing question-answering datasets.The framework itself is format-agnostic, so the question format is a design choice for ZEST rather than a theoretical requirement.
  • Task generation: Workers generated domain-general questions across U.S. presidents, dog breeds, and national parks, yielding approximately 100 base task descriptions per domain after deduplication.The selected domains contain many entities to which a single question can apply.
  • Task generation: Tasks were split into 50% test, 40% train, and 10% development, with derived paraphrase, semantic-flip, composition, and related variants kept in their corresponding splits.This preserves unique base questions in the test set alongside its derived generalization questions.
  • Answer annotation: Three workers annotated each task-document pair, using majority answers for yes/no tasks and the shortest answer contained within the others for extraction tasks.The resulting collection contains 25,026 task-input-answer triples covering 1,251 task descriptions.

5 Establishing a Baseline

The baseline study evaluates T5 and BART on ZEST using task-level competency metrics, consistency measures, and ablations. T5 performs substantially better than BART but solves only a small fraction of unseen tasks consistently, with errors concentrated in recognizing answers and handling NA cases.

  • Evaluation: Task competency uses F1 with partial-overlap scoring, while C@75 and C@90 require models to meet a threshold across complete tasks.For related paraphrase and semantic-flip tasks, C@T requires competence on both the base and changed descriptions.
  • Baseline Performance: 12% of unseen tasks reached 90% competency for the best T5 model on the test set, versus a 42% human performance estimate.The best model used T5 with multitask learning from other QA datasets.
  • Baseline Performance: 56% mean F1 on development data was achieved by T5-11B, while BART scored lower and multitask learning produced only slight improvements.The study reports mean F1 across instances separately from task-level competency.
  • Baseline Performance: Performance declined as generalization difficulty increased from Base to Output Structure, showing that consistently recovering tasks from descriptions remained difficult.The reported breakdown comes from the T5 model trained with multitask learning.
  • Annotation Artifacts & Ablations: Both question and context were needed for strong performance, while the context-only model predicted NA for every instance.Its nonzero F1 came from one task whose answers were all NA and therefore counted as competent by convention.
  • Error Analysis: Among 100 examined errors, 30% involved failing to recognize the answer and 37% involved predicting an answer when the gold label was NA.The model failed to produce parseable JSON on only 1.5% of structure questions and produced JSON for 0.008% of non-structure questions.

6 Conclusion

The paper presents learning from task descriptions as a framework for general-purpose NLP and introduces ZEST to test systematic generalization across unseen tasks. State-of-the-art performance reaches only 12%, while the framework also supports future few-shot extensions.

  • 6 Conclusion: The framework and ZEST dataset target NLP systems that solve tasks from natural-language descriptions and systematically generalize across four areas.ZEST is designed to evaluate whether models truly understand each task rather than merely answer individual instances.
  • 6 Conclusion: 12% is the state-of-the-art performance on ZEST, leaving substantial room for improvement.The result underscores the difficulty of learning complex unseen tasks from descriptions alone.
  • 6 Conclusion: The framework also permits few-shot settings that combine task descriptions with a handful of examples, enabling future application of meta-learning approaches.The authors identify this as an avenue for future work and make models, code, and data available.

B Data Preprocessing

The preprocessing pipeline converts each task description and document into a text-to-text input, while formatting complex answers as JSON when required.

  • B Data Preprocessing: Each model input prepends “zeroshot question: ” to the task description and “zeroshot context: ” to the document before joining them with whitespace.The same text-to-text format supports paired task descriptions and contexts.
  • B Data Preprocessing: Answers are formatted as JSON for output-structure generalization, so models produce either text or JSON depending on the question type.This supports more complex zero-shot relation-extraction tasks.

C Training Details

The experiments use released T5 and BART implementations, with T5 11B serving as the principal baseline model.

  • C Training Details: The appendix provides additional training and prediction details, with reproducibility code available for the baseline results.The code and experiments are released through the project data page.
  • C Training Details: The baselines build on T5 11B, a text-to-text encoder-decoder pretrained through masked language modeling and multitask learning.T5 11B contains 11 billion parameters and uses the original released implementation.

Training, Evaluation, & Hyper-parameters

The experiments use T5 and BART baselines with specified training configurations, hardware, and compute details. Evaluation-related metric distinctions and implementation references are also documented.

  • T5 Training: T5 models used fixed sequence lengths of 512 tokens, batch size 32, and teacher-forced text-to-text training.The ZEST + MTL baseline equally weighted component datasets and sampled them at the same rate.
  • Metric Note: The reported evaluation metric differs from the metric described in the referenced implementation note.This distinction concerns the more complex evaluation used for reporting results.
  • BART Training: BART-large used 406 million parameters, four decoding beams, and tuned learning rate, epochs, batch size, and sequence lengths.The best development model used learning rate 3e-5 for 15 epochs, with approximately 3.5 minutes of training per epoch on one RTX 8000 GPU.

D Evaluation Details

Evaluation measures whether models solve complete tasks using task-specific scoring for classifications, answer spans, and structured outputs. The procedure extends reading-comprehension conventions to novel output structures.

  • Evaluation Framework: The evaluation aggregates performance over all instances of each task to determine whether the model competently performs that task.The framework expands prior reading-comprehension evaluation conventions to account for novel output structures.
  • Evaluating Classification: Classification answers are scored with the modified F1 metric across the yes, no, and NA classes.This scoring follows the modified F1 definition used for task evaluation.
  • Evaluating Answer Spans: Answer-span evaluation aligns multiple predicted and gold spans, computes word-overlap F1, and takes the maximum score across annotator answers.The resulting span score is then used to compute the task F1 score.
  • Evaluating Output Structure Questions: Structured outputs align predicted and gold entities, score key-value pairs, and weight value scores by key F1 so incorrect keys receive penalties.All key-value pairs for a task contribute to the final task F1 score.

E Baseline Results

The appendix provides detailed baseline and human-performance breakdowns for ZEST, including T5 trained with multitask data, T5 trained only on ZEST, and BART-large.

  • Baseline Results: The appendix reports detailed results for T5 with multitask training, T5 with ZEST-only training, BART-large, and human performance.It also reproduces the multitask T5 results table for comparison.
  • T5 Multitask Results: Table 7 is the detailed result table for T5 trained with multitask data.Its caption identifies the table's scope but does not provide individual result values.
  • Single-Dataset Baselines: Table 8 contains detailed results for T5 trained only on ZEST, while Table 9 contains detailed BART-large results.The supplied passages identify these tables but do not state their cell values.
  • Human Performance: Table 10 provides the detailed human-performance results on ZEST.The supplied passage identifies the table without reporting its individual values.
Loading 2011.08115v1…