PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Text Classification: From Shallow to Deep Learning

Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S. Yu, Lifang He

arXiv:2008.00364v6cs.CL

TL;DR

Text classification research spans many models, datasets, and evaluation metrics, motivating an updated synthesis. The paper surveys approaches from traditional models to deep learning, organizes them into a taxonomy, and compares their datasets, metrics, and benchmark results. It concludes that traditional and deep models use different routes to improve performance, while robustness, interpretability, representation, and resource trade-offs remain challenges.

  • Problem

    The proliferation of text-classification methods, datasets, and evaluation metrics creates a need for a comprehensive and updated review.

  • Method

    The paper surveys traditional and deep learning models, text representations, datasets, evaluation metrics, and benchmark results, organizing models by their structures and applications.

  • Results

    Pre-trained models achieve better results on most datasets, while Naïve Bayes remains less computationally complex but requires manual classification features.

  • Takeaways & Limitations

    The survey provides comparative references for model selection across text-classification applications, datasets, and evaluation settings.

  • Takeaways & Limitations

    Text representation methods face a trade-off between semantic information loss in simple vector-space approaches and the time cost of semantically based methods.

Abstract

from arXiv · show

Text classification is the most fundamental and essential task in natural language processing. The last decade has seen a surge of research in this area due to the unprecedented success of deep learning. Numerous methods, datasets, and evaluation metrics have been proposed in the literature, raising the need for a comprehensive and updated survey. This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021, focusing on models from traditional models to deep learning. We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification. We then discuss each of these categories in detail, dealing with both the technical developments and benchmark datasets that support tests of predictions. A comprehensive comparison between different techniques, as well as identifying the pros and cons of various evaluation metrics are also provided in this survey. Finally, we conclude by summarizing key implications, future research directions, and the challenges facing the research area.

1 INTRODUCTION

Text classification assigns predefined labels to text and is important across NLP applications, but manual processing is slow, challenging, and vulnerable to human factors. This survey organizes developments from traditional feature-engineered models to deep learning, covering models, datasets, metrics, benchmarks, and challenges.

  • Text classification assigns predefined labels to text for applications including sentiment analysis, topic labeling, question answering, and dialog act classification.
  • Manual classification of large text collections is time-consuming and challenging, while fatigue and expertise can affect accuracy.
  • Traditional models depend on manually engineered features, whereas deep learning methods can extract features automatically.
  • Traditional approaches such as Naïve Bayes, K-Nearest Neighbor, and Support Vector Machine improved accuracy and stability over rule-based methods but remain costly to engineer.
  • The survey reviews text-classification models from traditional methods through deep learning and also covers datasets, evaluation metrics, benchmark results, and research challenges.

2 TEXT CLASSIFICATION METHODS

Text classification extracts features from raw text and predicts text categories. The surveyed methods range from traditional classifiers and representations to widely used deep learning models such as TextCNN and BERT.

  • Text classification extracts features from raw text and predicts the categories of text data.
  • Traditional methods include Naïve Bayes, K-Nearest Neighbor, Support Vector Machine, Random Forest, XGBoost, and LightGBM.
  • TextCNN is the most referenced deep learning model in the survey, while BERT is widely used for text classification because of its effectiveness across datasets.

2.1 Traditional Models

Traditional text classification pipelines preprocess text, represent it with engineered features, and apply classical models such as Naïve Bayes, KNN, SVM, decision trees, and random forests. These methods offer practical strengths but remain constrained by feature engineering and assumptions about textual data.

  • Traditional Models: Traditional pipelines preprocess raw text, create representations such as BOW, N-gram, TF-IDF, word2vec, or GloVe, and then classify the resulting features.Preprocessing generally includes word segmentation, data cleaning, and statistics.
  • PGM-based Methods: Naïve Bayes calculates posterior probabilities under a conditional independence assumption, simplifying computation but reducing performance when features are numerous or strongly correlated.Its simple structure and small parameter count support broad use in text classification.
  • KNN-based Methods: KNN assigns an unlabeled text to the most common category among its k nearest training samples, but it can be unusually slow on large-scale datasets.It classifies by estimating distances rather than building a separate model.
  • SVM-based Methods: SVM converts text classification into multiple binary tasks and selects a maximum-margin hyperplane, with kernel and feature choices supporting nonlinear classification.The optimization objective is associated with strong generalization and low classification error.
  • Tree and Integration-based Methods: Decision trees recursively divide datasets into subsets whose leaves represent categories, while random forests aggregate multiple tree classifiers trained with independently sampled random vectors.Decision-tree methods use construction and pruning stages; integration methods combine multiple classifiers.
  • Traditional Models: Across traditional models, feature engineering remains time-consuming and costly, although traditional approaches usually perform better than deep learning models on small datasets.These methods also commonly disregard sequential structure or contextual information in text.

2.2 Deep Learning Models

Deep learning models automate feature learning for text classification, progressing from MLPs and ReNNs to RNNs, CNNs, attention, and hybrid architectures.

  • DNNs learn high-level features from word vectors and are evaluated on downstream text-classification tasks.The reviewed models cover different dataset traits, evaluation metrics, and experimental datasets.
  • ReNN-based Methods: ReNNs recursively learn text semantics and syntax-tree structure without task-specific feature design.They combine word representations into parent nodes using a shared weight matrix.
  • MLP-based Methods: MLPs automatically capture features by connecting input, hidden, and output layers, while treating text as a bag of words.The reviewed three-layer structure uses weighted connections and achieves high performance on text-classification benchmarks.
  • RNN-based Methods: RNNs process embedded words sequentially to model historical information and long-range dependencies.Their recurrent computation uses word embeddings as inputs to successive RNN cells.
  • RNN-based Methods: RNN research includes topic-aware, adversarially trained, capsule-based, and other variants for contextual modeling and robustness.TopicRNN combines latent topics with RNNs, while VAT perturbs word embeddings to improve robustness and reduce overfitting.
  • CNN-based Methods: CNNs apply different kernels to multiple sequence chunks, filtering text features from multiple angles for classification.Unlike recurrent processing, convolution can operate on several sequence regions simultaneously.

2.2.4 CNN-based Methods.

CNN-based methods extract discriminative local features through convolution and pooling, while related work combines CNNs with embeddings, RNNs, and attention to capture broader context.

  • CNN-based Methods: TextCNN uses one convolution layer and max pooling to identify discriminative phrases while keeping word vectors static.It learns model hyperparameters other than the fixed word vectors.
  • CNN-based Methods: Text embeddings are organized at character, word, and sentence levels to address OOV words, learn word semantics, and capture sentence relationships.The levels differ according to the minimum embedding unit used for the text.
  • CNN-based Methods: CNN-RNN fusion models use convolution for feature extraction and recurrent encoding for contextual understanding.BLSTM-2DCNN combines BiLSTM with two-dimensional max pooling, while MTNA combines BiLSTM and CNN layers.
  • Attention-based Methods: Attention-based methods address the limited interpretability of CNN and RNN hidden representations in classification errors.Attention can improve performance while exposing the importance of text regions.
  • Attention-based Methods: Self-attention represents each word with query, key, and value vectors to capture long-range dependencies and compute outputs in parallel.The output weights depend on query-key similarity and value vectors.
  • Attention-based Methods: Attention models are applied to fine-grained aspect sentiment and question-answer matching tasks.They focus on sentence regions relevant to an aspect or capture dependencies between question and answer embeddings.

2.2.6 Pre-trained Methods.

Pre-trained language models learn general semantic representations from unlabeled text and transfer them to classification and other NLP tasks through architectures such as ELMo, GPT, BERT, and their extensions.

  • Pre-trained Methods: ELMo, GPT, and BERT differ in how they construct contextual representations and use bidirectional or staged pre-training objectives.ELMo uses bidirectional LSTM context, GPT combines unsupervised pre-training with supervised fine-tuning, and BERT jointly conditions on both contexts.
  • Pre-trained Methods: RoBERTa modifies BERT with dynamic masking and longer pre-training using more data.Its masking pattern is generated anew for each input sequence.
  • Pre-trained Methods: XLNet uses permutation-based autoregressive pre-training rather than BERT’s denoising autoencoder masking.It incorporates ideas from Transformer-XL while learning bidirectional context across factorization orders.
  • Pre-trained Methods: ALBERT reduces BERT’s parameterization through factorized embeddings and encoder parameter sharing, while changing its sentence objective.It replaces next-sentence matching with next-sentence ordering.
  • Pre-trained Methods: BART reconstructs text corrupted by noise, whereas SpanBERT masks contiguous spans and predicts them using boundary information.Both architectures are presented as extensions of pre-training for downstream NLP tasks.
  • Graph-based Methods: GNN-based models represent text as graphs to learn syntactic, co-occurrence, and neighboring-node information for classification and related tasks.Graph design can connect documents, sentences, and words, while TextGCN builds a heterogeneous graph for a whole dataset.

2.2.8 Others.

Additional approaches extend text classification with twin networks, adversarial regularization, reinforcement learning, memory, external knowledge, and quantum-inspired representations, while model families retain distinct trade-offs.

  • Other Models: Siamese neural networks use shared weights on two inputs to produce comparable representations for semantic similarity and classification.Applications include settings with variable-length sequences and unbalanced medical data.
  • Other Models: VAT regularizes semi-supervised models through local distributional smoothness and perturbations applied to word embeddings.The reviewed method improves robustness, generalization, and word-embedding performance with relatively few hyperparameters.
  • Other Models: Reinforcement learning can construct structured sentence representations by selecting task-relevant words and hierarchical structures.The approach combines Information Distilled LSTM and Hierarchical Structured LSTM representations.
  • Other Models: Memory networks combine inference with long-term memory for joint aspect and opinion extraction and topic-aware classification.Topic Memory Networks encode latent topic representations indicative of class labels.
  • Other Models: Other reviewed directions recast sentiment classification as question answering and add commonsense knowledge or quantum-inspired representations.These approaches target matching information, event interpretation, or joint semantic and sentiment representation.
  • Model Comparison: RNNs lack parallel computation, whereas CNNs parallelize feature extraction and can retain location information through improved designs.The survey also notes that deeper CNNs do not always improve performance because of parameter-optimization and pooling limitations.
  • Model Comparison: Transformers parallelize computation and efficiently model interactions between words, but their attention becomes computation-heavy for long sequences.The survey notes improved Transformer models aimed at reducing this complexity.

3.1 Datasets

The survey organizes datasets by text-classification application, covering sentiment analysis, news classification, topic labeling, question answering, and natural language inference.

  • Sentiment Analysis: Sentiment-analysis datasets support binary or fine-grained multi-class classification of subjective emotional text.Examples include MR, SST, MPQA, IMDB, Yelp, and Amazon reviews.
  • News Classification: News-classification datasets represent topic recognition and recommendation applications across multiple category granularities.The survey lists 20NG, AG News, R8, R52, and Sogou; AG News contains 120,000 training and 7,600 testing texts.
  • Topic Labeling: Topic-labeling datasets assign one or more subjects to documents, with resources spanning encyclopedic, biomedical, question-answering, and legal domains.Examples include DBpedia, Ohsumed, YahooA, EUR-Lex, Amazon670K, Bing, Fudan, and PubMed.
  • Question Answering: Extractive question answering uses text-classification models to select the correct answer from candidates.The survey describes SQuAD, TREC-QA, WikiQA, and MS MARCO as question-answering datasets.
  • Natural Language Inference: Natural language inference and paraphrase datasets evaluate whether one text’s meaning follows from another or whether sentence pairs are semantically equivalent.SNLI and MNLI use textual-entailment labels, while MSRP uses binary paraphrase labels.

3.2 Evaluation Metrics

The survey distinguishes evaluation metrics by task structure: accuracy and F1-family measures are central for classification, while ranking and multilabel tasks require specialized metrics.

  • Overview: Accuracy and F1 score are widely used for text classification, while P@K, Micro-F1, and MRR address multilabel and question-answering settings.Metric choice varies with task difficulty and structure.
  • Question-answering metrics: Exact Match measures whether a question-answering prediction exactly matches the ground-truth answer.It is identified as the primary metric used on SQuAD.
  • Question-answering metrics: Mean Reciprocal Rank evaluates ranking performance in question answering and information retrieval using the ground-truth answer’s rank.The survey defines rank(i) as the ranking of the ground-truth answer at answer i-th.
  • Multi-label metrics: Hamming-Loss measures misclassified instance-label pairs when a related label is omitted or an unrelated label is predicted.
  • Single-label metrics: For single-label classification, accuracy measures correct predictions, precision measures positive-prediction correctness, recall measures recovered positives, and F1 is their harmonic average.The survey also notes that accuracy, F1, and recall reach their desired value at 1.
  • Multi-label metrics: For multilabel classification, Micro-F1 aggregates label performance, Macro-F1 weights labels equally, and P@K or NDCG@K support extremely large label sets.The survey describes Micro-F1 as suitable for unbalanced distributions and notes Macro-F1’s sensitivity to classes with high precision and recall.

4 QUANTITATIVE RESULTS

The survey compares traditional and deep learning models on classic datasets and reports stronger results for pre-trained deep models on most datasets, while noting task-specific exceptions and trade-offs.

  • Evaluation setup: The comparison evaluates published model performance by classification accuracy on MR, SST-2, IMDB, Yelp, Amazon, 20NG, AG, DBpedia, and SNLI.Table 4 reports results from models’ original articles and identifies the most accurate result in bold.
  • Traditional models: 81.8% and 79.4% are the SST-2 accuracies reported for NB and SVM, respectively.NB outperforms SVM on this two-category dataset.
  • Model trade-offs: NB has lower computational complexity than deep models but requires manually designed features, making direct migration to other datasets difficult.
  • Deep learning models: Pre-trained models obtain better results on most datasets, with MR and 20NG noted as exceptions to the reported BERT-based experiments.The survey highlights BERT, RoBERTa, and XLNET as pre-trained models to try for text-classification tasks.
  • Deep learning models: 83.8% is the MR accuracy reported for RNN-Capsule, the best result identified for that dataset.The survey attributes this to category-specific capsules that can output sentiment-trend words without applying linguistic knowledge.

5 FUTURE RESEARCH CHALLENGES

The survey identifies future challenges in text classification across data, model design, semantic robustness, and interpretability. Key boundaries include dependence on labeled data, semantic information loss, adversarial vulnerability, and unclear model behavior.

  • Challenges from Performance Perspective: Improving accuracy does not establish human-level semantic understanding, and small noisy samples can substantially change confidence or reverse decisions.The survey calls for practical proof of semantic representation ability and robustness.
  • Challenges from Data Perspective: Text classification models remain dependent on numerous labeled examples, making zero-shot, few-shot, and specialized-domain settings difficult.Small or zero-sample settings can significantly affect performance, while annotating new datasets is time-consuming.
  • Challenges from Data Perspective: Adding external knowledge can improve semantic representation, but selecting what to add and fitting it within input-scale limits remains challenging.The survey discusses knowledge bases, knowledge graphs, conceptual information, and commonsense knowledge as possible sources.
  • Challenges from Data Perspective: Multi-label classification must preserve relationships among labels and rich document semantics despite lossy embedding and encoding.Reducing semantic loss during training remains an open problem for hierarchical and complex label information.
  • Challenges from Model Perspective: Efficient semantic text representations remain unresolved because vector-space methods lose semantic information while richer semantic methods are time-consuming.Deep-learning representations commonly use word embeddings, whose representation units vary across languages.
  • Challenges from Performance Perspective: Traditional and deep models generally perform well, but their resistance to interference and the interpretability of deep models require improvement.Adversarial samples can sharply reduce performance, while the optimization process lacks clear guidelines for explaining model behavior.

6 CONCLUSION

The paper surveys text classification from traditional models through deep learning, covering model mechanisms, datasets, evaluation metrics, benchmark results, and future challenges.

  • 6 CONCLUSION: The survey contrasts traditional models’ reliance on feature extraction and classifier design with deep models’ improvements in representation learning, structure, data, and knowledge.It includes summary tables for primary models, datasets, evaluation metrics, and leading-model results across applications.
  • 6 CONCLUSION: It organizes datasets and evaluation metrics for single-label and multi-label tasks and reports quantitative results for leading models on classic classification datasets.The paper concludes by summarizing possible future research challenges in text classification.
Loading 2008.00364v6…