Source-linked AI summary
Deep Learning Based Text Classification: A Comprehensive Review
Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, Jianfeng Gao
TL;DR
Text classification requires methods that can handle diverse tasks without relying on limited hand-crafted representations. This paper surveys deep-learning models and datasets, analyzes benchmark performance, and finds substantial improvements across text-classification tasks while identifying interpretability and robustness as open challenges.
Problem
Classical text-classification pipelines rely on hand-crafted features and cannot fully exploit large amounts of training data.
Method
The paper reviews more than 150 deep-learning text-classification models, summarizes more than 40 datasets, analyzes selected models on 16 benchmarks, and discusses future directions.
Results
Deep-learning models lead to significant improvements across the reviewed text-classification tasks, including around 78% relative reduction in classification error on SST-2.
Takeaways & Limitations
Model choice depends on task and domain characteristics, in-domain labels, and application latency and capacity constraints.
Takeaways & Limitations
The reviewed deep-learning models are generally not interpretable, and some researchers question their language understanding and robustness in mission-critical domains.
Abstract
from arXiv · showhide
Deep learning based models have surpassed classical machine learning based approaches in various text classification tasks, including sentiment analysis, news categorization, question answering, and natural language inference. In this paper, we provide a comprehensive review of more than 150 deep learning based models for text classification developed in recent years, and discuss their technical contributions, similarities, and strengths. We also provide a summary of more than 40 popular datasets widely used for text classification. Finally, we provide a quantitative analysis of the performance of different deep learning models on popular benchmarks, and discuss future research directions.
1 INTRODUCTION
Text classification assigns labels to textual units across applications, but classical approaches depend on hand-crafted features and domain expertise. This survey reviews deep learning models, datasets, benchmark performance, and open challenges across major text-classification tasks.
- Text classification assigns labels or tags to sentences, queries, paragraphs, and documents for applications including sentiment analysis, spam detection, and question answering.
- Automatic text classification is increasingly important as industrial text data scales, with rule-based and machine-learning approaches forming the main categories.
- Classical machine-learning pipelines extract hand-crafted features before classification, requiring tedious feature engineering and depending strongly on predefined representations.
- Neural approaches replace hand-crafted features with learned embeddings, progressing from early underperforming models to much larger pretrained models such as word2vec and ELMo.
- The paper also identifies limited interpretability and concerns about robustness and language understanding as continuing challenges for deep-learning models.
- The survey reviews more than 150 deep-learning models, more than 40 datasets, and selected benchmark results across sentiment analysis, news and topic classification, extractive question answering, and natural language inference.
2 DEEP LEARNING MODELS FOR TEXT CLASSIFICATION
This section surveys more than 150 deep learning models for text classification, organizing them by neural architecture. The models represent text as bags of words, sequences, trees, local patterns, capsules, or attention-based structures, with extensions for long-range dependencies and semi-supervised learning.
- Model Architecture Categories: The survey groups more than 150 models by architecture, including feed-forward, recurrent, convolutional, capsule, and attention-based networks.The categories are pedagogical rather than exclusive; architectures such as Transformers combine multiple components.
- Capsule Neural Networks: Capsule networks represent sentences or documents as vectors and use capsule layers to preserve or aggregate features beyond conventional pooling operations.Text capsule models include convolutional and fully connected capsule layers for predicting class labels.
- Feed-Forward Neural Networks: Feed-forward models average or sum word embeddings before classification, and simple DAN and fastText variants can capture useful text information.DAN can outperform more sophisticated compositional models on datasets with high syntactic variance, while fastText adds word n-grams for local order.
- RNN-Based Models: RNN-based models process word sequences to model dependencies, while LSTMs use memory cells and gates to address vanilla RNN gradient and long-term-dependency problems.Tree-LSTM extends LSTM to tree structures, and other variants add adaptive memory, multiple timescales, or latent topics.
- CNN-Based Models: CNN-based models detect local, position-invariant patterns, with DCNN using dynamic k-max-pooling to capture short- and long-range word and phrase relations.CNNs have become popular for text classification because key phrases can express sentiment or topic.
- Beyond Supervised Learning: Adversarial and virtual adversarial training perturb word embeddings to improve supervised or semi-supervised text classification, with mixed objectives reported to improve over supervised approaches.The mixed objective combines cross-entropy, adversarial, and virtual adversarial losses using labeled and unlabeled data.
3 HOW TO CHOOSE THE BEST NEURAL NETWORK MODEL FOR MY TASK
Choosing a text-classification architecture depends on the task, domain, label availability, and deployment constraints. The paper proposes a five-step recipe centered on pretrained language models, adaptation, task-specific design, fine-tuning, and compression.
- Architecture choice varies with the target task, domain, in-domain labels, and latency and capacity constraints.
- PLM Selection: Using pretrained language models leads to significant improvements across popular text-classification tasks, with autoencoding models often outperforming autoregressive models.Examples include BERT and RoBERTa as autoencoding models and OpenAI GPT as an autoregressive model.
- Domain adaptation: Domain adaptation can continually pre-train a general-domain model on in-domain data when the target domain differs substantially.For domains with abundant unlabeled text, pretraining a language model from scratch is also presented as an option.
- Task-specific model design: Task-specific layers transform contextual representations into final outputs, with their architecture selected according to the task’s linguistic structure.The paper contrasts feed-forward networks, RNNs, and CNNs according to the structures they capture.
- Fine-tuning: Task-specific layers may be trained with the PLM fixed or jointly with it, while multi-task fine-tuning can leverage labeled data from similar domains.
- Model compression: PLMs often require knowledge distillation or other compression methods to meet real-world latency and capacity constraints.
4 TEXT CLASSIFICATION DATASETS
The survey organizes widely used text-classification datasets by application, covering sentiment, news and topic classification, question answering, and natural language inference. These datasets vary substantially in task format, label structure, scale, and domain.
- The reviewed datasets are grouped by sentiment analysis, news categorization, topic classification, question answering, and natural language inference.
- Sentiment analysis: Sentiment datasets include binary and fine-grained labels across movie, review, opinion, and product data, with scales ranging from thousands to millions of samples.Examples include IMDb, Movie Review, SST, MPQA, Yelp, and Amazon.
- News and topic classification: News and topic datasets cover short articles, newsgroup documents, knowledge-base entries, medical abstracts, legal documents, scientific papers, and abstract sentences.Examples include AG News, 20 Newsgroups, Sogou News, Reuters, DBpedia, Ohsumed, EUR-Lex, WOS, and PubMed.
- Additional datasets extend coverage to topic classification and question answering, including PubMed 200k RCT, Irony, Twitter, arXiv, SWAG, and SelQA.
- Question answering: Question-answering datasets differ in source and answer format, including Wikipedia-derived pairs, real user queries, ranked answers, unanswerable questions, and paraphrase pairs.The surveyed datasets include SQuAD, MS MARCO, TREC-QA, WikiQA, and Quora.
5 EXPERIMENTAL PERFORMANCE ANALYSIS
The paper evaluates deep-learning text-classification models with task-appropriate metrics and compares them with classical or earlier baselines across 16 popular benchmarks. Across sentiment, news, topic, question-answering, and NLI tasks, deep learning produces substantial performance gains.
- 5.1 Popular Metrics for Text Classification: The analysis uses accuracy, error rate, precision, recall, F1, exact match, MRR, MAP, AUC, and related metrics for different evaluation settings.Accuracy and error rate satisfy Error Rate = 1 - Accuracy; precision and recall are especially useful for imbalanced test sets.
- 5.2 Quantitative Results: Deep-learning models lead to significant improvements across the evaluated text-classification tasks compared with non-deep-learning models used as prior state-of-the-art systems or baselines.
- Sentiment analysis: 78% relative reduction in classification error is reported on SST-2 after the introduction of the first deep-learning sentiment model.
- News and topic classification: News categorization and topic classification show a trend similar to sentiment analysis across AG News, 20-NEWS, Sogou News, DBpedia, and Ohsumed.
- Question answering: On SQuAD and WikiQA, the paper attributes the significant performance lift to the use of BERT.
- Natural language inference: SNLI and MNLI exhibit steady performance improvement over the five years covered by the analysis.
6 CHALLENGES AND OPPORTUNITIES
Despite major progress from deep learning, text classification still faces challenges involving data efficiency, difficult task coverage, interpretability, memory, and model compression.
- New Datasets for More Challenging Tasks: New datasets are needed for multi-step-reasoning question answering, multilingual documents, and extremely long documents.
- Modeling Commonsense Knowledge: Incorporating commonsense knowledge into deep learning models remains an open research direction, including for reasoning with incomplete information.
- Interpretable DL Models: Most deep learning models are not interpretable, leaving their learned behavior and performance differences across datasets insufficiently understood.Attention and self-attention provide some insight, but detailed analysis of model behavior and dynamics remains lacking.
- Memory Efficient Models: Modern neural language models require substantial memory for training and inference, creating compression needs for edge applications.Knowledge distillation and model compression are described as possible approaches.
- Few-Shot and Zero-Shot Learning: Few-shot and zero-shot learning remain important because most deep learning models require large amounts of domain labels.Fine-tuning pretrained language models requires fewer domain labels than training from scratch.
7 CONCLUSION
The paper surveys more than 150 deep learning models and more than 40 text classification datasets, analyzes performance on public benchmarks, and discusses open challenges and future directions.
- The survey covers more than 150 deep learning models developed over the past six years for text classification tasks.
- It reviews more than 40 popular text classification datasets and quantitatively analyzes model performance on several public benchmarks.
- The paper concludes by discussing open challenges and future research directions for deep learning-based text classification.
A DEEP NEURAL NETWORK OVERVIEW
The appendix introduces commonly used deep learning architectures for natural language processing and directs readers to a comprehensive discussion elsewhere.
- The overview introduces MLPs, CNNs, RNNs, LSTMs, encoder-decoders, and Transformers as commonly used deep learning models for NLP.
A.1 Neural Language Models and Word Embedding
Neural language models learn dense word representations from text, while word-level, subword, and contextual embeddings address different representation needs and limitations.
- Neural language models learn dense word embeddings in a self-supervised fashion for use in NLP applications.
- Word-Level Embedding: Word2vec maps words appearing in similar contexts to similar vector representations and supports simple algebraic operations in vector space.
- Word2vec has limitations involving out-of-vocabulary extension, word morphology, and word context, motivating improved embedding models.
- Word-Level Embedding: CBOW predicts the central word from context words, whereas Skip-Gram predicts context words from the central word.
- Word-Level Embedding: GloVe learns embeddings by performing matrix factorization on a word co-occurrence matrix.
- Subword Embedding: Subword and character embeddings can handle out-of-vocabulary words and reduce embedding model size.
- Contextual Embedding: Contextualized embeddings map a word to different vectors depending on its context, unlike Word2vec and GloVe.
A.2 Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM)
RNNs represent sequences through recurrent hidden states but struggle with long-term dependencies because of gradient vanishing and explosion. LSTMs address this limitation with a memory cell and gates that regulate information flow.
- RNNs: RNNs process sequential data by combining the current input with the previous hidden state to produce a new hidden state and optionally an output.The final hidden state, or a weighted average of hidden states, can represent the input sequence for downstream tasks.
- RNN limitations: RNNs cannot capture long-term dependencies in very long sequences because of gradient vanishing and explosion.
- LSTM: LSTM improves long-term dependency modeling through a memory cell and input, output, and forget gates that regulate information flow.The memory cell retains values over arbitrary time intervals.
- LSTM: The LSTM formulation relates word-embedding inputs and hidden states to gate values and memory-cell updates through learned parameters and element-wise operations.The forget gate determines whether to reset the memory cell, while the input and output gates control its input and output.
A.3 Convolutional Neural Networks (CNNs)
CNNs apply convolution, nonlinear activation, and pooling to text segments to extract local features and aggregate them into global features. They originated in computer vision and were later adopted for NLP applications.
- Origins and applications: CNNs were originally developed for computer vision and later applied to various NLP tasks.Fukushima proposed an early CNN-related model, and LeCun and colleagues popularized efficient back-propagation training.
- Architecture: CNNs use convolutional layers to apply sliding kernels over text segments and extract local features.
- Architecture: Nonlinear layers apply activation functions to local feature values, while pooling layers aggregate features through max-pooling or mean-pooling.Together, these operations form global features from local patterns.
A.4 Encoder-Decoder Models
Encoder-decoder models map an input to an output by first compressing the input into a latent representation and then reconstructing or predicting the output. They are widely used for sequence-to-sequence tasks, while autoencoders use identical inputs and outputs.
- Encoder-decoder process: Encoder-decoder models encode input x into a latent vector z and decode z into output y.The encoding and decoding stages are expressed as z = f(x) and y = g(z).
- Encoder-decoder process: The latent representation is expected to capture the underlying semantics of the input.
- Applications: Encoder-decoder models are widely used for sequence-to-sequence tasks such as machine translation.The illustrated example maps an English word sequence to its German translation.
- Autoencoders: Autoencoders are encoder-decoder models whose input and output are the same and can be trained unsupervised by minimizing reconstruction loss.
A.5 Attention Mechanism
Attention represents word importance through weights that identify relevant relationships during language modeling. It allows decoders to focus on source-sentence parts rather than relying only on a fixed-length state vector, while self-attention models relationships within the same sentence.
- Attention mechanism: Attention in language models uses importance weights to estimate how strongly words correlate with one another when predicting a word.
- Attention mechanism: Bahdanau attention lets the decoder search for source-sentence parts relevant to predicting each target word.This avoids compressing the entire source sentence into a single fixed-length state vector.
- Attention mechanism: Attention scores weight a linear combination of input-word hidden vectors to generate the decoder’s output.
- Visualization: In French-to-English translation, brighter attention cells indicate words with greater impact on the English translation.
- Self-attention: Self-attention learns correlations among words in the same sentence and is used in tasks including reading, summarization, and image captioning.Transformers also use self-attention.
A.6 Transformer
Transformers address the sequential-processing bottleneck of recurrent models by computing attention scores in parallel across words. Their architecture stacks encoder and decoder layers built from multi-head attention and position-wise feed-forward sub-layers.
- Transformers compute attention scores in parallel for every word to model the influence words have on one another.
- The Transformer consists of stacked encoder and decoder layers, each containing multi-head attention followed by a position-wise feed-forward network.
- Multi-head attention performs scaled dot-product attention multiple times for each set of queries, keys, and values.
- An optional attention mask prevents target-word information from being used improperly during attention computation.