PaperlayerView exact revision on arXiv

Source-linked AI summary

RAFT: A Real-World Few-Shot Text Classification Benchmark

Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, Michael Noetel, Andreas Stuhlmüller

arXiv:2109.14076v3cs.CLcs.AIcs.LG

TL;DR

Few-shot classification benchmarks have not been designed to measure progress on naturally occurring applied tasks. RAFT benchmarks such tasks under deployment-like evaluation, and its baselines show persistent challenges for models and human labelers alike.

  • Problem

    Existing few-shot benchmarks do not directly measure progress on applied classification tasks, which are economically valuable and may involve long texts or other real-world challenges.

    Source: §1 · pp. 1–2

  • Method

    RAFT benchmarks naturally occurring classification tasks using 50 public training examples, task instructions, and an evaluation setup that mirrors deployment.

    Source: §1 · pp. 2–3

  • Results

    Baselines reveal model struggles with long texts and many classes, and humans outperform GPT-3 on 8 of 11 tasks.

    Source: §1 · p. 3, §4.4 · p. 8

  • Takeaways & Limitations

    RAFT provides a benchmark for measuring the gap between research and practice on economically valuable classification tasks.

    Source: §6 · p. 10

    RAFT includes only English tasks, so additional languages are needed to capture the distribution of real-world tasks.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Large pre-trained language models have shown promise for few-shot learning, completing text-based tasks given only a few task-specific examples. Will models soon solve classification tasks that have so far been reserved for human research assistants? Existing benchmarks are not designed to measure progress in applied settings, and so don't directly answer this question. The RAFT benchmark (Real-world Annotated Few-shot Tasks) focuses on naturally occurring tasks and uses an evaluation setup that mirrors deployment. Baseline evaluations on RAFT reveal areas current techniques struggle with: reasoning over long texts and tasks with many classes. Human baselines show that some classification tasks are difficult for non-expert humans, reflecting that real-world value sometimes depends on domain expertise. Yet even non-expert human baseline F1 scores exceed GPT-3 by an average of 0.11. The RAFT datasets and leaderboard will track which model improvements translate into real-world benefits at https://raft.elicit.org .

1 Introduction

RAFT addresses the lack of few-shot classification benchmarks designed to measure progress on naturally occurring, applied tasks. It pairs realistic tasks and deployment-like evaluation with baselines that expose challenges for current systems and human labelers.

1 Introduction

  • RAFT focuses on classification because it supports high-quality automated evaluation, covers economically valuable tasks, and lacks existing real-world benchmarks.Source: §1 · p. 2
  • Existing few-shot classification benchmarks are not optimized to measure progress in applied settings and may omit challenges such as reasoning over long texts.Evaluation can also diverge from deployment through closed-book restrictions, uninformative labels, or extra validation examples.Source: §1 · p. 2
  • RAFT is designed to measure how recent and upcoming NLP advances benefit applications using naturally occurring, inherently valuable classification tasks.The tasks can require information retrieval, domain expertise, long-document parsing, and use of instructions.Source: §1 · p. 2
  • RAFT provides 50 public training examples per task and a larger unlabeled test set, keeps test labels private, and evaluates submissions through a leaderboard.Source: §1 · p. 3
  • Current techniques struggle with reasoning over long texts and tasks with many classes, while crowdsourced results show RAFT spans moderate-to-difficult tasks.Collecting human baselines was difficult on some datasets, particularly where domain expertise matters.Source: §1 · p. 3

2 Related Work

Prior work studies few-shot learning through prompting or fine-tuning, and existing benchmarks pursue goals such as transfer, diagnosing model weaknesses, or instruction following. RAFT instead aims to represent real-world classification tasks and their evaluation conditions.

2.1 Few-shot learning in NLP

  • In-context learning prompts a pretrained language model with labeled examples, while prompt-based fine-tuning trains it with masked-language modeling objectives on few examples.PET is another name for prompt-based fine-tuning; LM-BFF dynamically constructs prompts.Source: §2.1 · p. 3

2.2 Few-shot NLP benchmarks

  • FLEX, FewGLUE, CrossFit, and NaturalInstructions include classification tasks with meaningful textual labels.Source: §2.2 · p. 3
  • Unlike benchmarks targeting transfer, model weaknesses, or instruction following, RAFT selects tasks and evaluation to represent real-world classification.CrossFit excludes tasks requiring external sources, domain knowledge, or long documents; existing benchmarks do not allow open-domain retrieval.Source: §2.2 · p. 3

3 Benchmark Description

RAFT is a few-shot classification benchmark whose tasks, data selection, and evaluation rules are intended to mirror real-world classification problems. Its 11 selected datasets and benchmark rules retain practical task features while enabling automated comparison.

3 Benchmark Description

  • RAFT focuses on few-shot classification because automated evaluation is more reliable than for generation, and its tasks and evaluation are chosen to mirror real-world problems.Source: §3 · p. 4

3.1 Tasks

  • A RAFT classification task is a dataset of labeled natural-language entries with one-to-one natural-language class names and labeling instructions.Source: §3.1 · p. 4

3.1.1 Dataset selection criteria

  • Dataset selection favored naturally occurring tasks with intrinsic real-world value, retained imbalanced classes, and excluded tasks whose correct labels were trivially discoverable online.The value criterion is subjective, and the authors aimed to approximate the distribution of valuable classification tasks.Source: §3.1.1 · p. 4

3.1.2 Dataset preparation

  • For test sets larger than 5,000 examples, the authors randomly sampled 5,000; they also added textual labels when datasets lacked them.Source: §3.1.2 · p. 4

3.1.3 Selected RAFT datasets

  • RAFT selected 11 datasets, and its overview identifies long inputs, domain expertise, and detailed instructions as real-world challenges represented by the tasks.Source: §3.1.3 · p. 4, Table 1 · p. 5

3.2.1 Setting and rules

  • The evaluation uses 50 randomly chosen labeled examples per task, provides no extra validation examples, and supplies task instructions and textual labels.Transfer learning, further pre-training, and unlabeled test data are permitted.Source: §3.2.1 · pp. 5–6

3.3 Metrics

  • RAFT scores macro-averaged F1, including for binary datasets, and averages scores across datasets; its open-domain setting cannot restrict retrieval to information available before labeling.Source: §3.3 · p. 6

4 Baselines

The baselines compare GPT-3 and other automatic systems with crowdsourced humans under few-shot conditions. Humans generally outperform GPT-3, while results identify challenges involving many classes, long inputs, domain expertise, and zero-shot classification.

4.1.2 Classification

  • GPT-3 classifies by normalizing the probabilities assigned to each class’s first generated token, adding numeric prefixes when labels share that token.The baseline uses in-context examples and also evaluates a zero-shot version.Source: §4.1.2 · p. 7, §4.1 · p. 7

4.1.3 Parameter selection

  • Task-specific instructions improve averaged F1 by .04, semantic example selection by 0.03, and the best prompt example count varies by dataset.For long-input datasets, fewer detailed examples can work better; shorter inputs can fit more complete examples.Source: §4.1.3 · p. 8

4.2 Other automatic baselines

  • Other automatic baselines include GPT-Neo, GPT-2, zero-shot GPT-3 and BART, AdaBoost over word n-grams, and a plurality-class predictor.The neural baselines besides few-shot GPT-3 perform worse than AdaBoost; smaller generative models still outperform plurality.Source: §4.2 · p. 8, §4.4 · p. 9

4.3 Human baseline

  • Crowdsourced human baselines use qualification followed by annotation, with the plurality vote of five labelers on 100 randomly selected test examples.Workers receive the same 50 labeled examples, textual labels, and task-specific instructions as practitioners.Source: §4.3 · p. 8

4.4 Analysis

  • Humans outperform GPT-3 on 8 of 11 tasks, and the authors expect exceeding the crowdsourced baseline to require substantial model advances.GPT-3 struggles particularly with many classes, long inputs, and some domain-specific tasks.Source: §4.4 · p. 8
  • GPT-3’s largest human-performance gap occurs on B77, which has 77 classes, while long-input tasks such as NIS and OSE pose difficulties.With only 50 labeled examples, many B77 classes lack a corresponding labeled example; GPT-3’s context window may contribute to long-input difficulty.Source: §4.4 · pp. 8–9
  • Crowdsourced humans substantially outperform GPT-3 on only one of four identified domain-expertise tasks; zero-shot GPT-3 falls below plurality and BART is not competitive with few-shot models.GPT-3 beats humans on Over and TAI, while smaller neural baselines besides few-shot GPT-3 perform worse than AdaBoost.Source: §4.4 · p. 9

5 Discussion

The discussion identifies limits in RAFT’s English-only and subjectively selected dataset coverage, notes potential harms and costs, and outlines priorities for stronger baselines and more application-sensitive evaluation.

5.1 Limitations

  • RAFT covers only English tasks, and subjective dataset selection may introduce biases, with the authors’ organizations disproportionately in technology and policy.The authors say additional languages are needed to capture the distribution of real-world tasks more fully.Source: §5.1 · p. 9

5.2 Impact

  • Including hate-speech data may harm readers, while reliance on massive corporate transformer models risks widening the gap between individual teams and industry labs.The authors believe the benefits of studying hate-speech detection likely outweigh the disadvantages of publicizing such datasets.Source: §5.2 · p. 9

5.3 Future Work

  • Future work includes stronger human baselines, since some datasets require expertise such as law, and additional automatic baselines using prompt-based fine-tuning or open-domain retrieval.The authors expect retrieval-enabled models to surpass models that do not use retrieval.Source: §5.3 · p. 9
  • Future evaluation should account for application-specific priorities such as minimizing false positives versus false negatives, and explore learning from explanations alongside labeled examples.The authors report that task-specific instructions improved performance and pose example explanations as a possible extension.Source: §5.3 · p. 10

6 Conclusion

RAFT tests language models on economically valuable classification tasks across domains in the true few-shot setting. The authors present it as a deployment-mirroring benchmark that can help measure the research-practice gap and guide future benchmarks.

6 Conclusion

  • RAFT tests language models across multiple domains on economically valuable classification tasks in the true few-shot setting, aiming to mirror applied task distributions and evaluation.The authors describe it as a template for future benchmarks that mirror deployment.Source: §6 · p. 10

A.1 Dataset licensing

The appendix records dataset licenses, including several unlicensed datasets and datasets released under Creative Commons, MIT, or other stated terms.

A.1 Dataset licensing

  • ADE Corpus V2, Overruling, Terms of Service, TweetEval Hate, and Twitter complaints are listed as unlicensed.The curating authors bear responsibility in the case of rights violations.Source: §A.1 · p. 13
  • Banking77 and the NeurIPS impact statement risks dataset use Creative Commons Attribution 4.0; the latter derivative dataset is also licensed under that license.The original NeurIPS impact statement dataset has an MIT License.Source: §A.1 · p. 13
  • OneStopEnglish and TAI safety research use Creative Commons Attribution-ShareAlike 4.0, while semiconductor org types uses Creative Commons Attribution-NonCommercial 4.0.Source: §A.1 · p. 13
  • Systematic review inclusion uses Creative Commons Attribution 4.0 International.Source: §A.1 · p. 13

A.2 Dataset examples

The appendix points readers to Table 3 for one training example from each dataset.

A.2 Dataset examples

  • Table 3 presents one training example from each dataset.Source: §A.2 · p. 13

A.3 Task-specific instructions

The paper provides dataset-specific instructions for human annotators and adapts them for automatic baselines; Table 4 gives excerpts.

A.3 Task-specific instructions

  • Table 4 presents excerpts from the instructions for each dataset.Source: §A.3 · p. 13
  • The full human-annotator instructions are adapted for use with automatic baselines.The instructions are based on publications or consultation with dataset creators.Source: §A.3 · p. 13

ADE Corpus V2 (ADE)

The supplied passages describe the ADE labeling criteria and include a separate banking-query task prompt. ADE labels distinguish therapeutic drug mentions from adverse effects occurring as a consequence of drug intake.

ADE Corpus V2 (ADE)

  • The ADE prompt asks whether a sentence is related to an adverse drug effect.Source: ADE Corpus V2 (ADE) · p. 13
  • Drug or chemical mentions count only in a therapeutic context, excluding metabolites, reaction byproducts, and hospital chemicals.Names may include brand, trivial, abbreviated, or systematic forms.Source: ADE Corpus V2 (ADE) · p. 13
  • ADE labeling counts adverse effects only when they occur as a consequence of drug intake.Included effects range from signs and symptoms to diseases, organ damage, or death.Source: ADE Corpus V2 (ADE) · p. 13
  • A separate prompt describes classifying banking customer-service queries into 77 categories.Source: ADE Corpus V2 (ADE) · p. 13

NeurIPS impact statement risks (NIS)

The supplied passages specify a criterion for identifying harmful applications of research and note that Table 3 contains a labeled training example from every dataset.

NeurIPS impact statement risks (NIS)

  • A harmful-application label requires the statement to support harmful applications of the research being done, rather than a past risk the research addresses.Source: NeurIPS impact statement risks (NIS) · p. 13
  • Table 3 presents one training example from each dataset with its textual label.Source: Table 3 · p. 14

OneStopEnglish (OSE)

These parts document the classification tasks and labeling criteria, dataset purpose, collection and annotation procedures, limitations, and licenses. They also describe appendix methods for GPT-3, AdaBoost, and human baselines.

OneStopEnglish (OSE)

  • The section’s task prompts span article-level ESL classification, overruling-sentence detection, institution typing, systematic-review screening, hate-speech detection, and complaint detection.The review-screening rules include reproducible reviews of monetary charitable donations and exclude primary research, non-systematic reviews, cause-related marketing, and other prosocial behaviors.Source: OneStopEnglish (OSE) · p. 14, Overruling (Over) · p. 14, Semiconductor org types (SOT) · p. 15

TAI safety research (TAI)

  • TAI safety research requires substantive motivation by good outcomes for transformative AI, safety content, a researcher audience, and a low seriousness threshold; peer review is not required.Motivation and seriousness involve subjective judgments; speculative work errs toward inclusion, while work likely written without TAI-safety concerns is excluded.Source: TAI safety research (TAI) · p. 16

Terms of Service (ToS)

  • ToS guidance marks clauses shifting consumers’ jurisdiction away from their residence as unfair, non-resident governing law as potentially unfair, and broad liability limits as potentially unfair or clearly unfair.Liability clauses affirming non-excludable provider liabilities are fair; clauses limiting liability for physical injury, intentional damage, or gross negligence are clearly unfair.Source: Terms of Service (ToS) · pp. 16–17
  • ToS guidance treats unilateral-change and contract-by-use clauses as potentially unfair, and considers arbitration outside the consumer’s residence or based on an arbiter’s discretion clearly unfair.Fully optional arbitration is marked clearly fair; the supplied guidance also describes unilateral termination and content-removal clauses without specifying their labels.Source: Terms of Service (ToS) · p. 17

Hate speech against immigrants

  • Immigrant-targeted hate requires both an immigrant or refugee target and content that promotes hatred, violence, dehumanization, harm, or intimidation; if either condition is absent, label “No.”The guidance includes insults, threats, incitement, ethnic inferiority claims, and dehumanization as examples, while excluding hate speech against other targets and merely offensive language.Source: Hate speech against immigrants · pp. 17–18

Preprocessing, Cleaning, Labeling

  • NeurIPS impact statements were collected through software-based scraping and processing, with partial human validation; harmful-application labels used a three-annotator majority vote.Ought contractors performed the labeling, and the harmful-application subset was randomly sampled.Source: Collection · pp. 19–20, Preprocessing, Cleaning, Labeling · p. 20

A.4.1 NeurIPS impact statement risks

  • The NeurIPS dataset was assembled from 2020 proceedings PDFs converted to XML and supplemented with subject areas, authors, affiliations, and metadata; harmful-application labeling was contractor-performed.The original dataset was created to assess the 2020 impact-statement requirement and can support further analysis of that requirement and meta-analysis of accepted papers.Source: Motivation · p. 18, Preprocessing, Cleaning, Labeling · p. 20, Uses · p. 20

Uses

  • The NeurIPS impact-statement dataset may contain extraction errors and subjective labels, so its harmful-application labels warrant cautious interpretation.Some statements may be truncated or incorrectly extracted, and judgments about harm depend on labelers’ views; the labels roughly capture whether authors noted potential negative effects rather than only positive impacts.Source: Uses · p. 21

Composition

  • The semiconductor dataset is a sample of 500 organizations, and names containing “universit” were removed because models could classify them as universities without discrimination.Its human-created labels may be incorrect.Source: Composition · p. 21

Collection

  • A probabilistic sample of 500 organizations and conference-paper titles was collected through the IEEE API in March 2021 from ISSCC, VLSI, and IEDM contributions over 25 years.Duplicate organization names were removed; a student collected the data and was paid according to German law.Source: Collection · p. 22

Distribution

  • The semiconductor annotations are restricted to non-commercial research under CC BY-NC 4.0, whereas the TAI bibliography is released under CC BY-SA 4.0.The TAI bibliography had been used for a LessWrong report and a review, but not in academic papers; its creators cautioned against using it to create dangerous AI.Source: Distribution · pp. 22, 25, Uses · p. 24

Motivation

  • The TAI-safety bibliography was assembled to help donors assess organizations’ research output and to support future projects such as a living review.Jess Riedel and Angelica Deibel created it independently, with Riedel volunteering time and paying Deibel from personal funds.Source: Motivation · p. 22

Collection

  • The TAI-safety bibliography was compiled largely by hand using author and organization input, Google Scholar, and automated citation and Zotero tools; its creators report missing papers and subjective inclusion decisions.The dataset was collected haphazardly in 2019–2020, and its raw and processed data were not distinctly separated.Source: Collection · p. 23, Composition · p. 23, Preprocessing, Cleaning, Labeling · p. 24

A.5.1 Parameter selection

  • The GPT-3 parameter-selection runs used 20 training examples for the reported LOO comparisons, but TEH prompts mistakenly used 50 rather than the better-performing 25 examples.Other in-context baselines used the same prompt example count as GPT-3, which may be suboptimal for models with smaller context windows; improving these baselines was left to future work.Source: §A.5.1 · p. 25, Table 5 · p. 25, Table 6 · p. 25

A.6 AdaBoost baseline details

  • The AdaBoost baseline vectorized examples as counts of n-grams for n ∈[1, 5] and used 100 depth-3 decision trees with learning rate 1.0.Hyperparameter tuning found low tree depths and roughly 50–100 trees performed best, while reasonable learning-rate changes did not substantially affect results; tuning used leave-one-out cross-validation.Source: §A.6 · pp. 25–26

A.7.1 Labeling process

  • Human labelers qualified using 20 training examples and 30 references, then five labels per item were combined by plurality vote, with ties broken randomly.SRI received only an annotation phase of 200 points because of extreme class imbalance.Source: §A.7.1 · pp. 26–27

Related papers

Loading 2109.14076v3…