PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Classification via Large Language Models

Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, Guoyin Wang

arXiv:2305.08377v3cs.CL

TL;DR

LLMs using in-context learning underperform fine-tuned models on text classification because complex linguistic phenomena require stronger reasoning and in-context learning permits only limited demonstrations. CARP addresses these constraints with progressive clue-and-reasoning prompting and task-specific kNN demonstration retrieval, achieving leading benchmark results and strong low-resource performance.

  • Problem

    LLMs using in-context learning significantly underperform fine-tuned models on text classification because reasoning about complex linguistic phenomena and using limited demonstrations remain challenging.

  • Method

    CARP first extracts superficial clues, then induces diagnostic reasoning, and uses a fine-tuned model for kNN demonstration retrieval.

  • Results

    CARP achieves new SOTA performances on four of five benchmarks and comparable performance to SOTA on MR.

  • Takeaways & Limitations

    With 16 examples per class, CARP achieves performance comparable to supervised models trained with more than 1,000 examples per class.

  • Takeaways & Limitations

    CARP’s robustness to different hyper-parameters is reported with supporting experimental results placed in Appendix B.2.

Abstract

from arXiv · show

Despite the remarkable success of large-scale Language Models (LLMs) such as GPT-3, their performances still significantly underperform fine-tuned models in the task of text classification. This is due to (1) the lack of reasoning ability in addressing complex linguistic phenomena (e.g., intensification, contrast, irony etc); (2) limited number of tokens allowed in in-context learning. In this paper, we introduce Clue And Reasoning Prompting (CARP). CARP adopts a progressive reasoning strategy tailored to addressing the complex linguistic phenomena involved in text classification: CARP first prompts LLMs to find superficial clues (e.g., keywords, tones, semantic relations, references, etc), based on which a diagnostic reasoning process is induced for final decisions. To further address the limited-token issue, CARP uses a fine-tuned model on the supervised dataset for $k$NN demonstration search in the in-context learning, allowing the model to take the advantage of both LLM's generalization ability and the task-specific evidence provided by the full labeled dataset. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks, 97.39 (+1.24) on SST-2, 96.40 (+0.72) on AGNews, 98.78 (+0.25) on R8 and 96.95 (+0.6) on R52, and a performance comparable to SOTA on MR (92.39 v.s. 93.3). More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups. Specifically, using 16 examples per class, CARP achieves comparable performances to supervised models with 1,024 examples per class.

1 Introduction

LLMs using in-context learning underperform fine-tuned models on text classification because they struggle with complex linguistic phenomena and can use only a limited number of demonstrations. CARP is introduced to address these reasoning and evidence-access constraints.

  • LLMs with in-context learning significantly underperform fine-tuned models on text classification.
  • Text classification requires reasoning about concession, negation, intensification, irony, and other intricate linguistic phenomena.
  • Limited in-context-learning demonstrations prevent LLMs from using more than a small proportion of the training set.
  • CARP is introduced as an extensible approach for text classification.

SENTIMENT: Neutral

The supplied passages combine a sentiment-classification prompt with CARP’s progressive clue-and-reasoning strategy and its reported benchmark performance.

  • CARP decomposes text-classification reasoning into clue extraction, diagnostic reasoning, and final label determination.
  • CARP uses superficial clues such as keywords, tones, and semantic relations before inducing diagnostic reasoning.
  • CARP achieves new SOTA performances on four of five benchmarks and comparable performance to SOTA on MR.

2 Related Work

Related work contrasts pretrained language-model architectures and fine-tuning with in-context learning, while reviewing prompting, demonstration selection, reasoning, and text-classification methods.

  • 2.1 Large Language Models: LLMs include encoder-only, decoder-only, and encoder-decoder architectures, including BERT, GPT, and T5.
  • 2.1 Large Language Models: In-context learning generates textual responses from prompts containing a few annotated examples, unlike pretraining followed by task-specific fine-tuning.
  • 2.1 Large Language Models: Prior work studies continuous prompt optimization, in-context example selection, explanations, and chain-of-thought reasoning.
  • 2.3 Text Classification: Text classification assigns predefined labels such as sentiment polarity or topic to an input text.
  • 2.3 Text Classification: Earlier text-classification systems extract features with neural models and feed them into a classifier to obtain the final label.
  • 2.3 Text Classification: Recent in-context-learning methods reformulate inputs, compose sub-prompts with logic rules, and retrieve similar examples or label-word demonstrations.

3 Prompt Construction

CARP constructs prompts from task descriptions, demonstrations, and test inputs, while selecting demonstrations through task-specific kNN retrieval rather than only random or general-semantic similarity.

  • 3.1 Overview: Prompt-based in-context learning transforms text classification into generating a predefined textual response conditioned on a prompt.
  • 3.2 Prompt Construction: A CARP prompt contains a task description, annotated demonstrations, and the test input.
  • 3.2 Prompt Construction: The task description specifies the classification task, such as assigning positive or negative sentiment.
  • 3.2 Prompt Construction: Demonstrations provide decision-making evidence and an output format, and are required only for few-shot learning.
  • 3.3 Demonstration Sampling: Random sampling selects k training examples without guaranteeing semantic relatedness to the test sequence.
  • 3.3 Demonstration Sampling: kNN sampling retrieves demonstrations similar to the test sequence from the training set.
  • 3.3 Demonstration Sampling: SimCSE retrieves semantically similar examples but does not necessarily retrieve examples with matching labels.
  • 3.3 Demonstration Sampling: CARP uses a model fine-tuned on the training dataset as its kNN encoder, combining LLM generalization with task-specific evidence from the full labeled dataset.

4 Clues Collecting and Reasoning

CARP uses progressive clue collection, diagnostic reasoning, and final decision-making to address linguistic phenomena in text classification. In few-shot prompting, retrieved demonstrations include generated clues and reasoning.

  • Progressive reasoning: CARP decomposes classification into clue collection, reasoning, and decision-making stages.The model first gathers local evidence, then combines it into a diagnostic reasoning process before assigning a label.
  • Clue collecting: Clues include keywords, phrases, context, semantic meaning and relationships, tones, and references.For example, “snappy,” “clever,” and “want to hate it” provide sentiment clues in an opinion snippet.
  • Reasoning: Reasoning goes beyond superficial keywords by considering negation, intensification, irony, and other linguistic phenomena.The LLM treats collected clues and the input as premises for deeper justification.
  • Few-shot prompting: In few-shot prompting, each retrieved demonstration is augmented with its text, generated clues, reasoning, and gold label.The paper generates clues and reasoning for training examples, then concatenates them when examples are selected as demonstrations.
  • Prompt structure: The resulting prompt makes the LLM output clues first, reasons second, and the final decision last.This format supports progressive inference and allows clue and reasoning content to be prepared for demonstrations.
  • Voting: Majority vote selects the most frequent prediction, whereas weighted probability vote uses summed prediction probabilities across runs.These strategies address the diverse responses generated by LLMs under multiple sampling runs.

5 Experiments

Experiments compare CARP with prompting and supervised baselines on five classification datasets, including full-data, low-resource, and domain-adaptation settings. CARP outperforms CoT and supervised baselines broadly, while retaining strong performance with limited or out-of-domain training data.

  • Experimental setup: Experiments evaluate five datasets—SST-2, R8, R52, AGNews, and MR—under full-training and low-resource setups.Low-resource conditions sample 16, 128, 256, 512, or 1,024 instances per class.
  • Demonstration retrieval: Few-shot setups consistently outperform zero-shot setups, and fine-tuned RoBERTa representations retrieve better demonstrations than SimCSE representations.SimCSE retrieval also outperforms random sampling, showing the value of test-relevant demonstrations.
  • Reasoning strategies: CARP outperforms CoT by +1.48 on SST-2, +0.97 on AGNews, +2.76 on R8, +3.29 on R52, and +0.47 on MR.CoT itself outperforms vanilla prompting, which directly asks the LLM to generate labels without additional reasoning.
  • Full-training results: 97.39 on SST-2 (+1.24), 96.40 on AGNews (+0.72), 98.78 on R8 (+0.25), and 96.95 on R52 (+0.6) establish new SOTA results, while MR reaches 92.39 versus 93.3 SOTA.Single CARP also outperforms fine-tuned RoBERTa on all benchmarks.
  • Low-resource results: Using 16 examples per class, CARP achieves performance comparable to supervised models trained with 1,024 examples per class.With 512 examples per class, CARP is comparable to supervised models trained on the full dataset.
  • Domain adaptation: Switching demonstrations from SST-2 to Yelp on SST-2 test reduces CARP performance by only -0.5%, from 96.80% to 96.29%.The corresponding supervised-model comparison shows a larger -7.2% decrease, from 95.99% to 88.78%.

6 Ablation Studies

The ablations examine how demonstrations, label words, clues, ordering, and generated reasoning affect CARP’s text-classification performance. They find benefits from more demonstrations, annotation label words, clue use, low-to-high ordering, and sufficiently reliable generated reasoning.

  • Demonstrations: Increasing the number of demonstrations improves performance for both vanilla prompting and CARP on SST-2.The comparison uses Figures 3 and 4.
  • Label words: Annotation words as label words achieve the best performance among the evaluated label-word types, while flipped words cause a significant decrease.The evaluated types include position indices, annotation words, synonyms, flipped words, random words, and special tokens.
  • Clues: Keywords and phrases play the key role in SST-2 and R8 clue ablations.The study removes keywords and phrases, contextual information and semantic meaning, semantic relationships, tones, and references from clue-collection prompts.
  • Demonstration order: Low-to-high demonstration ordering achieves the best performance compared with random and high-to-low ordering, showing that performance is sensitive to ordering.Low-similarity demonstrations appear first, placing higher-similarity examples closer to the test sequence.
  • Reasoning quality: Generated reasoning has reliability above 95% for SST-2 and R5, perplexity below 4, and logic-faithfulness scores above 93%.These evaluations use sampled training instances and assess reliability, fluency, and entailment-based faithfulness.

7 Conclusion

The paper concludes that CARP improves text classification with strong benchmark results and low-resource and domain-adaptation capabilities. It proposes exploring CARP on additional natural language understanding tasks.

  • 7 Conclusion: CARP achieves new state-of-the-art performances on four of five widely used text-classification benchmarks.The conclusion identifies low-resource and domain-adaptation setups as additional areas of strong performance.
  • 7 Conclusion: Future work will explore CARP on more natural language understanding tasks.

A Dataset

The evaluation uses SST-2, R8, R52, AGNews, and MR datasets covering sentiment, news, and multi-class Reuters classification. Their documented sizes and sources define the benchmark settings.

  • Datasets: The benchmark suite comprises SST-2, R8, R52, AGNews, and Movie Review datasets.R8 and R52 are Reuters subsets with 8 and 52 classifications, respectively.
  • Dataset sources: SST-2 uses the original Rotten Tomatoes snippets and the train/dev/test splits from Socher et al. (2013).
  • Dataset sizes: R8 contains 5,485 training and 2,189 testing documents, while R52 contains 6,532 training and 2,568 testing documents.
  • Dataset sizes: AGNews contains 30,000 training and 1,900 testing examples for each class.

B.1 Fine-tuning Hyper-parameters

The fine-tuning experiments use RoBERTa and RoBERT-GCN on four NVIDIA 3090 GPUs with FP16, tuning optimization settings on a validation set.

  • B.1 Fine-tuning Hyper-parameters: RoBERTa and RoBERT-GCN are fine-tuned on four NVIDIA 3090 GPUs with FP16.
  • B.1 Fine-tuning Hyper-parameters: Validation-set tuning varies learning rate across 2e-5, 3e-5, and 4e-5 and batch size across 16 and 32.The fixed settings include dropout 0.3, weight decay 0.01, and warmup proportion 0.01.

B.2 The influence of hyper-parameters

This section investigates how temperature and frequency penalty affect Instruct-GPT3 on SST-2, with temperature varied from 0 to 1.0. It also presents example prompts for the setups in Section 3.

  • The experiments examine temperature and frequency penalty using Instruct-GPT3 on SST-2.
  • Temperature: Temperature τ controls generated-text variety, ranging from greedy decoding near zero to greater variety at higher values.The study varies τ from 0 to 1.0 while keeping topp=1.
  • The appendix provides example prompts for the setups described in Section 3.
Loading 2305.08377v3…