PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification

Martin Juan José Bucher, Marco Martini

arXiv:2406.08660v2cs.CLcs.AI

TL;DR

Although prompt-based generative AI offers an alternative to fine-tuning with labeled task data, its text-classification performance relative to fine-tuned LLMs remains an open question. The authors systematically compare ChatGPT-3.5, ChatGPT-4, Claude Opus, and Facebook’s BART with several smaller fine-tuned LLMs; fine-tuned models are evaluated on held-out test sets using Accuracy, Precision, Recall, and F1-scores. Fine-tuned smaller BERT-style models significantly outperform zero-shot generative AI models across all four applications when moderate amounts of training data are provided.

  • Problem

    Although prompt-based generative AI offers an alternative to fine-tuning with labeled task data, its text-classification performance relative to fine-tuned LLMs remains an open question.

    Source: §1 · p. 1

  • Method

    The authors systematically compare ChatGPT-3.5, ChatGPT-4, Claude Opus, and Facebook’s BART with several smaller fine-tuned LLMs; fine-tuned models are evaluated on held-out test sets using Accuracy, Precision, Recall, and F1-scores.

    Source: §1 · p. 1, §4 · p. 6

  • Results

    Fine-tuned smaller BERT-style models significantly outperform zero-shot generative AI models across all four applications when moderate amounts of training data are provided.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that smaller fine-tuned LLMs remain preferable for text classification, and provide a toolkit to make this approach more accessible.

    Source: §1 · p. 2

    Few-shot prompting requires careful example selection and systematic prompt evaluation, increasing time and costs and potentially eroding generative AI’s simplicity.

    Source: §6 · p. 13

Abstract

from arXiv · show

Generative AI offers a simple, prompt-based alternative to fine-tuning smaller BERT-style LLMs for text classification tasks. This promises to eliminate the need for manually labeled training data and task-specific model training. However, it remains an open question whether tools like ChatGPT can deliver on this promise. In this paper, we show that smaller, fine-tuned LLMs (still) consistently and significantly outperform larger, zero-shot prompted models in text classification. We compare three major generative AI models (ChatGPT with GPT-3.5/GPT-4 and Claude Opus) with several fine-tuned LLMs across a diverse set of classification tasks (sentiment, approval/disapproval, emotions, party positions) and text categories (news, tweets, speeches). We find that fine-tuning with application-specific training data achieves superior performance in all cases. To make this approach more accessible to a broader audience, we provide an easy-to-use toolkit alongside this paper. Our toolkit, accompanied by non-technical step-by-step guidance, enables users to select and fine-tune BERT-like LLMs for any classification task with minimal technical and computational effort.

1 Introduction

This paper compares prompt-instructed generative AI models with smaller fine-tuned LLMs for text classification, amid mixed early evidence about which approach performs better. Across four applications, fine-tuned models significantly outperform zero-shot models when moderate training data are available, and the paper provides a toolkit to make fine-tuning easier to use.

1 Introduction

  • The paper presents prompt-based generative AI as a simple alternative to fine-tuning, while noting that early empirical evidence on their relative classification performance was inconclusive.Traditional fine-tuning typically requires labeled task data and deep-learning code from end-users.Source: §1 · p. 1
  • The study systematically compares ChatGPT-3.5, ChatGPT-4, and Claude Opus with Facebook’s BART and several smaller fine-tuned LLMs.The paper argues that model size and complexity are no sufficient substitute for application-specific training data.Source: §1 · pp. 1–2
  • The comparison covers sentiment, approval or disapproval, emotion, and party-position classification across news, tweets, and political speeches.The four case studies include New York Times economic coverage, Kavanaugh-nomination tweets, German political texts, and nationalist parties’ European-integration positions.Source: §1 · p. 2
  • Fine-tuned smaller BERT-style models significantly outperform zero-shot ChatGPT and Claude Opus across all four applications when moderate training data are available.The advantage is especially pronounced on specialized, non-standard classification tasks.Source: §1 · p. 2
  • The toolkit is a Hugging Face-based Jupyter Notebook for selecting and fine-tuning smaller pre-trained LLMs for classification tasks and text categories.It supports multiple languages and binary or non-binary classification, includes class-imbalance methods and default hyperparameters, and comes with step-by-step guidance.Source: §1 · p. 2

2 Related Work and Contribution

Prior comparisons of fine-tuned and prompt-based models have reached conflicting conclusions, and some focus on tasks other than text classification. This paper adds a systematic comparison across tasks, text categories, current generative AI models, fine-tuned models, training-data sizes, and an accessible toolkit.

2 Related Work and Contribution

  • The paper’s focus is text classification, whereas some prior comparisons address other text-comprehension tasks or compare only fine-tuning methods or zero-shot approaches.The authors describe their work as an up-to-date, systematic empirical analysis of fine-tuned versus prompt-based models for classification.Source: §2 · p. 2
  • Earlier studies provide conflicting evidence on whether fine-tuned or prompt-based models perform better on text classification.Studies [15] and report advantages for fine-tuning, while [46] and [18] suggest ChatGPT can match or surpass fine-tuned models in some settings.Source: §2 · p. 2
  • The paper systematically compares performance across diverse classification tasks and text categories to identify where differences between approaches are most pronounced.Its comparison includes GPT-4 and Claude Opus, alongside several fine-tuned LLMs to clarify how model performance varies by task.Source: §2 · pp. 2–3
  • The authors analyze how training-data size affects fine-tuned model performance to inform how many labeled examples may be needed.The paper frames this as an ablation study relevant to its findings in favor of fine-tuned LLMs.Source: §2 · p. 3
  • The paper offers a one-stop-shop text-classification toolkit intended to make pre-trained LLMs usable by a wider audience.The toolkit is described as modular and intended to support additional models.Source: §2 · p. 3

3 Non-technical Background: From Keywords to Large Language Models

Text-as-data methods differ in how they represent language, from explicit human rules and word counts to representations that capture meaning across text sequences. The section explains how these choices shape the potential of machine learning and LLMs for classification.

3 Non-technical Background: From Keywords to Large Language Models

  • Hand-coding uses human coders and explicit codebooks, while dictionaries automate keyword-to-category mapping; machine learning instead derives rules from labeled examples.Hand-coding can be high-quality but labor-intensive and less replicable; dictionaries are fast and transparent but can be context-dependent and noisy.Source: §3 · p. 3

3.1 Bag-Of-Words

  • Bag-of-words represents documents as word-frequency rows, but ignores word order, grammar, and relations between word meanings.The resulting document-by-word matrix can be sparse and have more word columns than document rows (P > N).Source: §3.1 · p. 4

3.2 Word Embeddings

  • Word embeddings represent words as vectors that capture meaning and similarity, offering a more compact representation than bag-of-words.They can generalize to related unseen words and encode application-relevant relations, but still abstract from sentence structure and word order.Source: §3.2 · pp. 4–5

3.3 Pre-trained Large Language Models

  • Pre-trained LLMs represent sequences rather than individual words; BERT-style models gain task-specific orientation through fine-tuning, while GPT-style models generate answers from prompts.The attention mechanism lets LLMs account for negation and related structures; the authors argue that GPT-style models may lack sufficient examples of many classification tasks in pre- and post-training.Source: §3.3 · pp. 5–6

4 Method: Fine-tuned LLMs vs. Zero-Shot Generative AI Models

The study compares smaller fine-tuned LLMs with zero-shot generative AI models and BART across four heterogeneous text-classification case studies. Fine-tuned models are evaluated on held-out test data, while zero-shot models label the full datasets using task-specific prompts.

4 Method: Fine-tuned LLMs vs. Zero-Shot Generative AI Models

  • The comparison spans four case studies varying in classification concept, text form, language, and number of classes.The tasks include sentiment, stance, and emotion in English and German news, tweets, and speeches, with binary and multi-class settings.Source: §4 · p. 6
  • Five fine-tuned models—RoBERTa Base, RoBERTa Large, DeBERTa V3, ELECTRA Large, and XLNet—are run with the toolkit’s default hyperparameters.The authors use defaults to show the setup’s capabilities without requiring hyperparameter tuning.Source: §4 · p. 6
  • Fine-tuned models are trained on each case study’s training set and evaluated on a held-out test set using Accuracy, Precision, Recall, and F1-scores.Each model is trained three times with different random seeds, and the reported metrics are the mean and standard deviation across runs.Source: §4 · p. 6
  • Zero-shot models are prompted to label the full datasets, rather than only the test sets, with prompts describing the dataset, categories, and labeling task.Their generated labels are evaluated against the original human annotations using the same performance metrics as for fine-tuned models.Source: §4 · pp. 6–7
  • Across case studies, evaluated models include MAJ-VOT, which assigns every observation to the majority class, and RoBERTa Base (125M), RoBERTa Large (355M), DeBERTa (435M), and ELECTRA Large (335M).On imbalanced datasets, this baseline can suggest good performance on metrics sensitive to class imbalance, such as Accuracy, while Recall or F1-Score reveal poor performance for minority classes.Source: §4 · pp. 7–8

5 Results

Across the four case studies, fine-tuned models beat prompt-based systems: prompts remained competitive on sentiment but struggled on specialized stance, emotion, and EU-position tasks. The final analysis finds gains from moderate training data that level off thereafter.

5.1 Sentiment Analysis on The New York Times Coverage of the US Economy

  • On US-economy sentiment, ROB-LRG and DEB-V3 were strongest; prompted models also performed well but remained behind the fine-tuned models.Most other fine-tuned models except XLNET-LRG achieved values around 0.90, with small variances.Source: Table 1 · p. 8, §5.1 · p. 9

5.2 Stance Classification on Tweets about Kavanaugh Nomination

  • For Kavanaugh-tweet stance, DEB-V3 and ROB-LRG were strongest, and every fine-tuned model surpassed the prompt-based approaches.DEB-V3 and ROB-LRG scored 0.94 and 0.92, respectively, across metrics; BART, ChatGPT, and Claude scored only marginally above the naive majority-vote baseline.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 Emotion Detection on Political Texts in German

  • For German political-text anger detection, XLNET-LRG led, while BART, ChatGPT, and Claude performed substantially worse than the naive classifier.ELECTRA performed almost identically on German text and its English translation.Source: §5.3 · p. 10, Table 3 · p. 10

5.4 Multi-Class Stance Classification on Parties’ EU Positions

  • For EU-position classification, DEB-V3 outperformed the other models, while BART, ChatGPT, and Claude did not exceed the naive baseline.DEB-V3 scored above 0.90 for most metrics; most other fine-tuned models performed at least decently.Source: §5.4 · p. 11, Table 4 · p. 11

5.5 Fine-Tuning: The Effect of Training Set Size on Model Performance

  • Fine-tuned performance rose quickly with training data, then saturated, with the sweet spot generally between 200 and 500 observations.Below 200 observations, performance remained below potential; above 500, it began to level off. ROB-LRG outperformed BART on all datasets once training exceeded 200 observations. On imbalanced datasets, low-sample accuracy could be misleading because models predicted the majority class, while Macro-F1 showed the saturation pattern.Source: §5.5 · pp. 11–12, Figure 4 · p. 12

6 Discussion

Fine-tuned smaller LLMs remain superior in the reported classification tasks, while the discussion surveys possible future developments and practical production trade-offs.

Current findings

  • Fine-tuned smaller LLMs remain superior to zero-shot generative models across the applications, though future developments may shift the comparison.The advantage is more pronounced on non-standard tasks; the current findings assume moderate fine-tuning data.Source: §6 · pp. 12–13

Data Augmentation

  • Data augmentation can reduce fine-tuning data requirements by synthetically expanding labeled examples through back translation or synonym replacement.The cited work reports performance gains from back translation [35].Source: §6 · pp. 12–13

Secondary Pre-training

  • Domain-specific secondary pre-training can improve downstream classification by adding knowledge of domain vocabulary and usage.The passage describes gains for RoBERTa after further pre-training before classification fine-tuning [20].Source: §6 · p. 13

Model Architecture

  • Architecture research explores multimodality, mixture-of-experts designs, and better training-data quality as alternatives to relying on model scale alone.Prior work reports saturating improvements when scaling from hundreds of billions to trillions of parameters [34] [7].Source: §6 · p. 13

Prompt Engineering and Few-Shot Learning

  • Few-shot prompting can boost instruction-tuned models, but example selection and prompt evaluation increase time and costs and may erode its simplicity.Further architectural development could eventually allow few-shot prompting to surpass fine-tuned encoder-style models.Source: §6 · p. 13

Production considerations

  • Small fine-tuned models offer local operation and visible parameters, while professional generative-AI use still requires labeled validation data.Local storage and execution provide control and privacy advantages compared with transferring information to proprietary cloud-based models.Source: §6 · p. 13

7 Conclusion

The conclusion compares fine-tuned smaller LLMs with zero-shot generative models, reports results across the case studies, and presents a modular toolkit for broader use.

7 Conclusion

  • The comparison covers text categories, tasks, languages, and imbalanced datasets, including fine-tuned models such as RoBERTa.Source: §7 · p. 13
  • Fine-tuned smaller LLMs significantly outperform zero-shot generative models in all case studies, with performance beginning to saturate after around 200 training examples.Exploratory few-shot prompting did not improve GPT-3.5 or GPT-4 results on the study datasets.Source: §7 · p. 13
  • The authors provide a modular text-classification toolkit requiring no NLP or deep-learning domain knowledge.Users can add models, including models for other languages or newer releases made available through Hugging Face.Source: §7 · p. 13

A: Zero-Shot Prompts for GPT-3.5, GPT-4, and Claude Opus

The appendix describes how prompts and API settings were adjusted to obtain one valid class label for each text sample.

A: Zero-Shot Prompts for GPT-3.5, GPT-4, and Claude Opus

Sentiment Analysis on The New York Times Coverage of the US Economy

The sentiment prompt asks the model to classify a text snippet and return only one of the supplied labels.

Sentiment Analysis on The New York Times Coverage of the US Economy

Stance Classification on Tweets about Kavanaugh Nomination

The prompts define three zero-shot classification setups: stance toward Kavanaugh’s nomination, anger in German political texts, and parties’ positions on EU leave demands.

Stance Classification on Tweets about Kavanaugh Nomination

Emotion Detection on Political Texts in German

Multi-Class Stance Classification on Parties’ EU Positions

B: BART

The BART setup uses the BART Large MNLI model with Hugging Face’s Zero-Shot Classification pipeline, which applies natural language inference to select a class label.

B: BART

  • The setup uses facebook/bart-large-mnli with Hugging Face’s Zero-Shot Classification pipeline.Source: B: BART · p. 17
  • For each possible label, natural language inference compares the text to classify as a premise with that label as a hypothesis.Source: B: BART · p. 17
  • The pipeline evaluates each label’s hypothesis against the premise and outputs the label with the highest confidence.Source: B: BART · p. 17
Loading 2406.08660v2…