PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Few-Shot Learning Without Prompts

Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, Oren Pereg

arXiv:2209.11055v1cs.CL

TL;DR

Existing few-shot methods can be difficult to use because of prompt variability and large model requirements. SetFit fine-tunes Sentence Transformers contrastively before training a classifier on their embeddings. It achieves strong few-shot results with smaller models and is reported as faster, prompt-free, multilingual, and useful for distillation.

  • Problem

    Existing few-shot approaches can require large language models, specialized infrastructure, and manually generated prompts whose engineering affects outcomes.

  • Method

    SETFIT contrastively fine-tunes a Sentence Transformer on sentence pairs, then trains a classification head on embeddings from the fine-tuned model.

  • Results

    SETFIT outperforms the FINETUNE, PERFECT, and ADAPET baselines in the reported comparisons and matches or exceeds T-FEW 3B depending on sample size.

  • Takeaways & Limitations

    SETFIT provides a prompt-free few-shot classifier using smaller base models, with reported advantages in training and inference speed and applications to multilingual data and distillation.

  • Takeaways & Limitations

    Comparing computational costs across SETFIT, PET, and PEFT is not straightforward because their hardware and memory requirements differ.

Abstract

from arXiv · show

Recent few-shot methods, such as parameter-efficient fine-tuning (PEFT) and pattern exploiting training (PET), have achieved impressive results in label-scarce settings. However, they are difficult to employ since they are subject to high variability from manually crafted prompts, and typically require billion-parameter language models to achieve high accuracy. To address these shortcomings, we propose SetFit (Sentence Transformer Fine-tuning), an efficient and prompt-free framework for few-shot fine-tuning of Sentence Transformers (ST). SetFit works by first fine-tuning a pretrained ST on a small number of text pairs, in a contrastive Siamese manner. The resulting model is then used to generate rich text embeddings, which are used to train a classification head. This simple framework requires no prompts or verbalizers, and achieves high accuracy with orders of magnitude less parameters than existing techniques. Our experiments show that SetFit obtains comparable results with PEFT and PET techniques, while being an order of magnitude faster to train. We also show that SetFit can be applied in multilingual settings by simply switching the ST body. Our code is available at https://github.com/huggingface/setfit and our datasets at https://huggingface.co/setfit .

1 Introduction

Few-shot methods address label-scarce settings but can depend on large models, specialized infrastructure, and manually crafted prompts. SETFIT offers a prompt-free alternative based on Sentence Transformers and performs strongly across few-shot classification tasks.

  • Few-shot learning targets label-scarce scenarios where annotation is time-consuming and costly.
  • Existing ICL, PEFT, and PET approaches can require large language models, specialized infrastructure, and manual prompt engineering.
  • SETFIT is a simple, prompt-free Sentence Transformer method that does not require large-scale PLMs for high accuracy.
  • With 8 labeled Customer Reviews examples, SETFIT is competitive with full-data fine-tuning despite using a model three times smaller.
  • SETFIT is evaluated across few-shot text-classification datasets, including distillation and non-English settings, against standard, PET-, PEFT-, and prompt-free baselines.
  • The authors publicly release the code and data used in the work.

2 Related Work

Related work spans prompt-based prediction, parameter-efficient adaptation, and sentence-embedding approaches. SETFIT builds on Sentence Transformers while fine-tuning them for few-shot classification without relying on task prompts.

  • Sentence Transformer research uses sentence embeddings for text classification, while SETFIT fine-tunes the encoder in a Siamese manner for few-shot objectives.
  • In-context learning generates predictions from prompted examples without parameter updates but relies on GPT-3-scale models and substantial computational resources.
  • PEFT methods freeze most model parameters and update inserted feed-forward networks, but T-FEW remains too large for some practical uses.
  • PET converts classification into a masked-language-modeling cloze task using manually crafted prompts and label mappings.
  • PERFECT removes task prompts and verbalizers through task-specific adapters and multi-token label embeddings.

3 SetFit: Sentence Transformer Fine-Tuning

SETFIT trains a Sentence Transformer contrastively on constructed sentence pairs, then fits a classifier to embeddings from the fine-tuned encoder. At inference, the encoder and classifier produce predictions for unseen sentences.

  • Sentence Transformers use Siamese or triplet structures to produce dense embeddings that separate semantically similar and distant sentences.
  • SETFIT first fine-tunes a Sentence Transformer contrastively on sentence pairs, then trains a classification head on the resulting embeddings.
  • ST fine-tuning: Positive pairs contain sentences from the same class, while negative pairs combine sentences from different classes.
  • ST fine-tuning: The contrastive dataset concatenates positive and negative pairs across classes, with |T| = 2R|C| and R = 20 in the evaluations unless stated otherwise.
  • ST fine-tuning: For binary classification with K labeled examples, unique sentence pairs can yield K(K −1)/2 contrastive examples, exceeding the original K.
  • Classification head training: A logistic-regression head uses one embedding and class label per original training sample, and predicts from embeddings of unseen inputs.

4 Experiments

The experiments evaluate multiple SETFIT model sizes across varied text-classification datasets and compare them with standard fine-tuning and prominent few-shot baselines. Evaluation uses repeated training splits and reports average performance with variability.

  • The evaluation covers text-classification datasets with different task types and numbers of classes, using development data for SETFIT hyperparameter selection.
  • SETFIT models: Three SETFIT variations use underlying Sentence Transformer models of different sizes.
  • Baselines: The baselines include standard transformer fine-tuning, ADAPET, PERFECT, and T-FEW.
  • Baselines: ADAPET converts textual inputs into cloze-style questions aligned with masked-language-modeling objectives.
  • Baselines: T-FEW provides 11- and 3-billion-parameter versions, but the 11-billion model requires an 80GB A100 GPU and was not run.
  • Experiments use 10 random training splits per dataset and sample size, reporting average measures and standard deviations across splits.

5 Results

SETFIT substantially improves over several few-shot baselines with limited data, remains competitive with much larger prompt-based models, and trades some advantage as supervision increases.

  • 19.3 points: SETFITMPNET significantly outperforms FINETUNE at N = 8, with the gap decreasing to 5.6 points at N = 64.The comparison uses labeled training samples per class, with higher scores indicating stronger performance.
  • 13.6 and 2.6 points: SETFITMPNET outperforms PERFECT at N = 8 and N = 64, respectively.
  • 4.0 and 1.5 points: SETFITMPNET outperforms ADAPET at N = 8 and N = 64, respectively.
  • 5 points: at N = 64, SETFITMPNET outperforms T-FEW 3B on average while being prompt-free and more than 27 times smaller.At N = 8, SETFITMPNET is on par with T-FEW 3B.
  • 8.6 and 1.7 points: SETFITROBERTA outperforms GPT3 and PET on RAFT, respectively, and surpasses the human baseline in 7 out of 11 tasks.SETFITROBERTA falls short of T-FEW 11B by 4.5 points but is more than 30 times smaller and more efficient in training and inference.

6 Multilingual Experiments

The multilingual experiments evaluate SETFIT across six-language Amazon reviews under monolingual, English-transfer, and pooled-data settings. With N = 8 examples per class, SETFIT outperforms FINETUNE and ADAPET across all reported settings.

  • Experimental Setup: The multilingual experiments use MARC, a corpus of Amazon reviews in English, Japanese, German, French, Spanish, and Chinese labeled on a 5-star scale.The corpus was chosen to examine generalizability across typologically diverse languages.
  • Experimental Setup: SETFIT uses paraphrase-multilingual-mpnet-base-v2, trained on parallel data in over 50 languages, as its underlying multilingual model.
  • Method: The three settings are each, training and evaluating monolingually; en, training in English and evaluating each language; and all, training and evaluating on pooled data.
  • Method: The evaluation uses N = 8 samples per class and compares few-shot methods with fine-tuning on the full training set of 20,000 examples.
  • Results: SETFIT significantly outperforms FINETUNE and ADAPET in all MARC settings, with the best average performance from training on English data only.Table 4 reports MAE, where a higher value indicates weaker performance.

7 SETFIT Model Efficiency

SETFIT is evaluated as a compact student model for few-shot distillation, where its sentence-transformer architecture is trained against a larger SETFIT teacher using limited labeled and additional unlabeled data. It substantially outperforms a standard transformer student with little unlabeled data and offers large computational savings over T-FEW.

  • Motivation: SETFIT achieves state-of-the-art few-shot results with base models containing 110M and 355M parameters, while motivating more efficient models for deployment.The paper identifies computational efficiency as important when deployment cost and sustainability are prioritized.
  • Experimental setup: The SETFIT student uses 15M parameters and is compared with a same-size MiniLM transformer baseline, using 16 labeled samples per class plus varying unlabeled data.SETFITMPNET serves as the 110M-parameter teacher; the baseline student is MiniLM-L3-H384-uncased.
  • Method: The student learns from teacher-generated cosine similarities between sentence pairs, then trains its classification head on student embeddings and teacher logits.This distillation procedure applies the sentence-transformer representation to both pairwise training and classification.
  • Distillation results: 24.8, 25.1, and 8.9 average accuracy points separate SETFIT from the baseline at N = 8 on AGNews, Emotion, and SST5, respectively.The performance gains decrease as unlabeled data increases and become comparable at N = 1K.
  • Computational costs: SETFITMPNET is approximately an order of magnitude faster than T-FEW for inference and training, while SETFITMINILM is two orders faster with a 3.1-point average-score reduction.The comparison uses FLOPs-per-token estimates under fixed sequence lengths, training steps, and batch size.
  • Computational costs: 30 seconds and $0.025 per split are reported for SETFITMPNET training, compared with 700 seconds and $0.7 for T-FEW 3B.SETFITMPNET used a p3.2xlarge instance with 16GB GPU memory, whereas T-FEW 3B required at least 40GB.

8 Conclusion

The paper concludes that SETFIT combines efficient few-shot classification with prompt-free operation and applicability beyond English. It also reports usefulness for few-shot distillation and substantially lower computational requirements than comparable approaches.

  • Conclusion: SETFIT is introduced as a few-shot text classification approach with advantages over T-FEW, ADAPET, and PERFECT.The conclusion emphasizes speed, smaller effective base models, and avoidance of external compute requirements.
  • Efficiency: SETFIT is much faster at inference and training and can perform with smaller base models without requiring external compute.These advantages are stated as central practical benefits relative to comparable approaches.
  • Scope: SETFIT avoids the instability and inconvenience of prompting and remains effective across non-English languages with varying typologies.The multilingual claim is presented alongside the prompt-free property in the conclusion.
  • Distillation: SETFIT has also proven useful in few-shot distillation setups.This conclusion follows the paper’s evaluation of SETFIT as a student model.

A.1 Datasets

The appendix lists datasets used for development and testing across sentiment, topic, spam, toxicity, question, and counterfactual classification tasks. The collection spans multiple domains and label structures.

  • Dataset use: Development and test datasets are separately used for setting SETFIT hyperparameters and evaluating robustness across text types and class counts.The datasets are available through the SETFIT organization on the Hugging Face Hub.
  • News datasets: BBC News contains five topic classes, while AG News contains four classes covering World, Entertainment, Sports, and Business.Both datasets provide multiclass news classification settings.
  • Additional tasks: The appendix also includes Enron Spam, Student Question Categories, TREC-QC, and Toxic Conversations.These datasets represent spam detection, subject classification, question answering, and toxicity attributes.
  • Sentiment datasets: The appendix identifies Amazon Polarity as a binary-sentiment dataset of Amazon reviews collected over 18 years.Examples are labeled positive or negative.
  • Classification datasets: The datasets include SST5, Customer Reviews, Emotion14, AG News, SST2, IMDB, BBC News, and Amazon Polarity.These datasets cover sentiment, emotion, news-topic, and review classification.
  • Counterfactual data: Amazon Counterfactual contains professionally labeled binary counterfactual-detection examples, and the experiments use its English subset.Counterfactual statements denote events that did not happen or cannot happen.

A.2 ADAPET Training Procedure

The ADAPET procedure uses fixed training schedules and manually specified prompt patterns, with verbalizers adapted across datasets and languages. The appendix documents the templates and label choices used in these experiments.

  • Training schedule: ADAPET is trained for 1,000 batches, with development-set evaluation and checkpointing disabled when no development data is assumed.The final model state after 1,000 batches is used in that setting.
  • Verbalizers: English verbalizers are built from dataset label and label-text columns, while multilingual verbalizers are translated by native speakers.For multilingual data, labels are also mapped to star ratings and translated.
  • Prompt sources: Emotion is the only dataset with existing P3 prompts; prompts for Enron Spam, CR, and SST5 are adapted from related datasets.The adaptations make minimal changes for domain or label differences.
  • Prompt construction: Amazon Counterfactual prompts are manually generated because no relevant P3 prompts exist.The prompts follow standard prompt-creation practices described in P3.
  • Templates: The listed templates vary between factuality, counterfactuality, sentiment, and movie-review formulations, with answer choices matching each task’s labels.SST5 templates use five sentiment choices, while counterfactual templates use binary choices.
Loading 2209.11055v1…