PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Classification Algorithms: A Survey

Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura E. Barnes, Donald E. Brown

arXiv:1904.08067v5cs.LGcs.AIcs.CLcs.IRstat.ML

TL;DR

Text classification requires suitable methods for representing and categorizing increasingly complex documents, but selecting effective structures and techniques remains challenging. The paper surveys feature extraction, dimensionality reduction, classification algorithms, and evaluation methods, while examining their limitations and applications. It concludes that system selection must account for differences among techniques across pipeline stages and the characteristics of the application and data.

  • Problem

    Selecting suitable text representations and classification techniques remains challenging because bag-of-words models neglect semantics and word order, while coherent document meaning is not fully retained.

  • Method

    The paper provides a structured survey of text-classification preprocessing, feature extraction, dimensionality reduction, classifiers, evaluation methods, applications, and limitations.

  • Results

    The survey compares available techniques across pipeline stages and identifies limitations affecting representation, classification, and evaluation.

  • Takeaways & Limitations

    Choosing an efficient text-classification system requires understanding similarities and differences among techniques at different pipeline steps.

  • Takeaways & Limitations

    Word-embedding models retain sentence-level syntactic and semantic information but do not fully preserve the meaning of coherent documents.

Abstract

from arXiv · show

In recent years, there has been an exponential growth in the number of complex documents and texts that require a deeper understanding of machine learning methods to be able to accurately classify texts in many applications. Many machine learning approaches have achieved surpassing results in natural language processing. The success of these learning algorithms relies on their capacity to understand complex models and non-linear relationships within data. However, finding suitable structures, architectures, and techniques for text classification is a challenge for researchers. In this paper, a brief overview of text classification algorithms is discussed. This overview covers different text feature extractions, dimensionality reduction methods, existing algorithms and techniques, and evaluations methods. Finally, the limitations of each technique and their application in the real-world problem are discussed.

1. Introduction

The paper frames text classification as a four-phase pipeline—feature extraction, dimensionality reduction, classifier selection, and evaluation—and surveys methods and applications across these stages.

  • Pipeline overview: Text classification systems are organized into feature extraction, dimensionality reduction, classifier selection, and evaluation.The paper uses this pipeline to structure its discussion of technical implementations.
  • Feature extraction: Feature extraction converts cleaned, unstructured text into structured features using methods such as TF-IDF, TF, Word2Vec, and GloVe.Cleaning removes unnecessary characters and words before formal feature extraction.
  • Dimensionality reduction: Dimensionality reduction can reduce the time and memory costs caused by text data sets containing many unique words.The paper discusses PCA, LDA, NMF, random projection, autoencoders, and t-SNE.
  • Classification techniques: The survey covers traditional, ensemble, non-parametric, support-vector, tree-based, graphical, and deep learning classifiers.Examples include Rocchio, boosting, bagging, KNN, SVM, decision trees, random forests, CRFs, and deep learning approaches.
  • Evaluation: Evaluation methods include Fβ Score, Matthews Correlation Coefficient, ROC, and AUC, while accuracy is unsuitable for unbalanced data sets.The paper presents evaluation as the final pipeline stage.
  • Applications and limitations: Choosing an effective system depends on the application goal and data set because feature extraction techniques are not uniformly efficient.The paper also discusses limitations, real-world applications, and comparisons across pipeline stages.

2. Text Preprocessing

Text preprocessing converts unstructured documents into features for classification by cleaning noise, tokenizing text, and applying normalization or representation methods. The section also highlights that feature choices can lose word order, semantics, or document-level coherence.

  • Preprocessing overview: Text preprocessing cleans documents and applies feature extraction to produce informative representations for classification.The paper discusses removing noise before formal feature extraction and distinguishes weighted-word from word-embedding techniques.
  • Tokenization: Tokenization breaks text streams into words, phrases, symbols, or other meaningful units called tokens.The example sentence is represented as a sequence of individual tokens.
  • Text cleaning: Stopword removal, lowercasing, slang conversion, punctuation removal, and spelling correction address common textual noise.Lowercasing can conflate terms such as “US” and “us,” while spelling correction remains optional.
  • Normalization: Stemming and lemmatization consolidate word variants by modifying forms or mapping words to basic forms.The paper gives “studying” → “study” as a stemming example and describes lemmatization as obtaining a word’s lemma.
  • Weighted-word representations: Bag-of-words and n-gram features are simple representations, but bag-of-words loses word order and treats vocabulary terms as independent dimensions.N-grams can capture more information than 1-grams, while bag-of-words models also face scalability challenges with large vocabularies.
  • Word embeddings: Word embeddings map words or phrases to real-valued vectors and can represent similarity that independent word-index features miss.The survey focuses on Word2Vec, GloVe, and FastText, while noting that contextualized representations make vectors depend on word context.
  • Document-level limitations: Per-sentence feature extraction can fail to preserve coherent document meaning and cross-sentence references.The example identifies “This day” with “July 4th” and “She” with “Maryam” across different sentences, relationships that sentence-level extraction fails to retain.

3. Dimensionality Reduction

Dimensionality reduction addresses the time and memory costs of high-dimensional text features through linear, nonlinear, matrix-factorization, projection, and neural methods. The section surveys their objectives and applications in text processing.

  • Text feature spaces can make preprocessing expensive because documents contain many unique words and consume substantial time and memory.
  • PCA identifies an approximately data-containing subspace by finding uncorrelated variables that maximize preserved variance.It can preprocess data before supervised learning, reduce noise, and help avoid over-fitting.
  • LDA reduces dimensions using class information, with transformations based on ratios involving between-class, within-class, and overall variance.
  • NMF compresses a non-negative data matrix V into WH when (n + m)r < nm, producing a lower-dimensional representation.The surveyed NMF pipeline extracts terms, constructs weighted document vectors, applies NMF, and projects documents into r-dimensional space.
  • Random projection, autoencoders, and t-SNE provide nonlinear or unsupervised approaches for reducing or visualizing high-dimensional text features.Random projection targets high-volume feature spaces, while autoencoders reduce n dimensions to p with p < n.

4. Existing Classification Techniques

The survey presents established text-classification families, including Rocchio, ensemble methods, and traditional and neural approaches. Classifier selection is treated as the central pipeline decision.

  • The survey also covers logistic regression, Naïve Bayes, KNN, SVMs, tree-based methods, and neural networks for text classification.
  • Rocchio represents classes with prototype vectors built from TF-IDF-weighted training documents and assigns documents by Euclidean distance to centroids.
  • Boosting adapts training distributions according to previous classifier performance, whereas bagging uses bootstrap samples without examining previous classifiers.

endif

This section continues the survey of classification models with ensemble and probabilistic approaches. It also identifies practical limitations involving interpretability and assumptions about the data.

  • Bagging combines classifiers trained on different bootstrap samples and predicts the class selected most often by their outputs.
  • Boosting and bagging methods have computational-complexity and interpretability limitations, making feature importance difficult to discover.
  • Logistic regression is a linear classifier that predicts probabilities rather than classes and can support binary and multinomial classification.
  • Logistic regression prediction requires each data point to be independent.

4.4. Naïve Bayes Classifier

Naïve Bayes is presented as a generative text-classification baseline with known distributional and data-scarcity limitations. The section also introduces KNN and its dependence on storage and meaningful distance functions.

  • Naïve Bayes predicts document classes from Bayes-based probabilities and is widely used as a word-level baseline in document categorization.
  • Multinomial Naïve Bayes estimates word likelihoods from word counts and class-conditional probabilities, including smoothed probability calculations.
  • Naïve Bayes performs poorly on unbalanced data sets and assumes a particular data-distribution shape, while data scarcity complicates likelihood estimation.
  • KNN classifies a test document by scoring candidate classes from the similarities and labels of its k nearest training documents.
  • KNN is easy to implement and handles multiclass feature spaces, but large searches require substantial storage and meaningful distance functions.

4.6. Support Vector Machine (SVM)

SVM supports binary and multi-class text classification through linear, nonlinear, hierarchical, and multiple-instance formulations. The section also describes string-based feature mappings and limitations involving transparency and computational complexity.

  • Core formulations: SVM was originally designed for binary classification but is also used for multi-class problems through techniques such as One-vs-One and All-vs-One.One-vs-One constructs N(N −1) classifiers, while multi-class SVM can also formulate all k classes jointly.
  • Core formulations: Linear and nonlinear SVM classifiers separate 2D data, while text applications typically involve thousands of dimensions.The figure distinguishes class 1, class 2, and misclassified points by color.
  • Feature representations: String kernels map sequences into feature spaces, with spectrum kernels representing strings by counting word occurrences.String kernels are discussed for text, DNA, and protein classification.
  • Limitations: SVM string-sequence classification is limited by time complexity, while high dimensionality can reduce transparency in text-classification results.The cited discussion links string-kernel complexity to dictionary size and feature count.
  • Specialized variants: Stacking SVM applies individual classifiers hierarchically in a top-down category tree and generally produces more accurate results than single-SVM models.The described hierarchy contains multiple levels, such as domains and sub-domains.

4.7. Decision Tree

Decision trees classify data through hierarchical feature-based decomposition and attribute selection. Random forests extend this approach with parallel tree ensembles, trading fast training for slower prediction as the forest grows.

  • Decision trees: Decision trees hierarchically decompose the data space by selecting attributes to organize categorized data points.The section identifies choosing parent and child attributes as a central design challenge.
  • Decision trees: Information gain is used to choose the attribute with the largest reduction in entropy as the parent node.Candidate attributes divide the training set into subsets before their remaining entropy is evaluated.
  • Limitations: Decision trees are very fast for learning and prediction but are sensitive to small data perturbations and can be easily overfit.Validation and pruning may mitigate these effects, although the discussion characterizes their use as a grey area.
  • Random forests: Random forests generate parallel random decision trees and assign predictions by voting across the trained forest.The method is described as an ensemble learning technique for text classification.
  • Random forests: Random forests train quickly on text data compared with deep learning but predict more slowly as the number of trees increases.Reducing the forest size is described as a way to obtain faster prediction.

4.9. Conditional Random Field (CRF)

Conditional random fields combine graphical modeling with classification to model label sequences conditioned on observations. Their main stated drawbacks are computationally expensive training and inability to handle unseen words.

  • Model structure: CRFs are undirected graphical models that represent the conditional probability of a label sequence Y given observations X.The model uses potential functions over graph cliques and a normalization term.
  • Model structure: CRFs combine compact multivariate modeling with high-dimensional feature spaces, making them suitable for text data.The section explicitly attributes their usefulness for text to the high feature space.
  • Model structure: Each clique potential contributes to the probability of a variable configuration, with feature weights defining the potential function.The potential is expressed using a weight vector and a feature vector.
  • Limitations: CRF training has high computational complexity, especially for text datasets with large feature spaces.The section identifies this as the most evident disadvantage of CRFs.
  • Limitations: CRFs do not perform with unseen words that were absent from the training data sample.This limitation is stated alongside the training-complexity drawback.

4.10. Deep Learning

The section surveys neural architectures for text and document classification, including feedforward, recurrent, convolutional, generative, and hierarchical-attention models. It describes their structures and highlights issues involving gradient instability and high-dimensional text feature spaces.

  • Deep neural networks: DNNs learn relationships between vectorized text inputs and targets through connected hidden layers and use back-propagation for training.The described input may use TF-IDF, word embeddings, or another feature representation, while multi-class outputs use Softmax.
  • Recurrent networks: RNNs assign greater weights to previous sequence elements, supporting semantic analysis for text and sequential-data classification.The section notes that RNNs commonly use LSTM or GRU architectures.
  • Limitations: RNNs are vulnerable to vanishing and exploding gradients when gradient descent errors are back-propagated through the network.The limitation is stated for the extended RNN architecture.
  • Recurrent networks: LSTM uses multiple gates to regulate information entering node states and preserve long-term dependencies more effectively than basic RNNs.The section presents LSTM as particularly useful for overcoming vanishing gradients.
  • Convolutional networks: CNNs use convolutional feature maps and max pooling to identify discriminative phrases in text before fully connected layers.The section describes CNNs as commonly used for hierarchical document classification.
  • Recurrent networks: GRUs simplify LSTM architecture by using two gates without internal memory or a second nonlinearity.The gates are described as update and reset mechanisms.
  • Convolutional networks: CNN text classification can face very high dimensionality because the feature-space channel count may reach 50 K.The section contrasts this with image applications that generally have only three RGB channels.
  • Hierarchical attention: Hierarchical attention networks encode and attend to words at a lower level and sentences at an upper level for document classification.The architecture separates word encoding and attention from sentence encoding and attention.

Adam Optimizer

The paper surveys text-classification architectures and describes HDLTex as a hierarchical approach that specializes deep-learning models for different document-hierarchy levels. It also notes interpretability, data requirements, and theoretical understanding as important limitations of deep learning.

  • Hierarchical Deep Learning for Text (HDLTex): HDLTex addresses hierarchical document classification by specializing deep-learning architectures for each level of the document hierarchy.The model targets the performance decline of traditional multi-class classification as the number of hierarchically organized classes increases.
  • Hierarchical Deep Learning for Text (HDLTex): HDLTex combines parent-level and child-level models, with child-level models providing document inputs to the parent level.Figure 22 depicts the parent-level model above the child-level models Ψi.
  • Combined Architectures: Combined architectures such as RCNN and C-LSTM integrate recurrent, convolutional, and long-term dependency modeling for text classification.RCNN captures contextual information recurrently and constructs text representations with CNNs, while C-LSTM learns phrase-level features before modeling long-term dependencies.
  • Limitations: Deep learning models are limited by weak interpretability and insufficient theoretical understanding of how learned outputs are produced.The paper characterizes deep-learning methods as black boxes and notes that neural-network connection weights do not provide comprehensive explanations of the modeling process.
  • Limitations: Deep learning generally requires more data than traditional machine-learning algorithms, limiting its use for classification on small data sets.The paper identifies the larger data requirement as a limitation of deep-learning classification methods.

5. Evaluation

The paper reviews evaluation metrics for text classifiers and emphasizes that inconsistent data collection, train/test splits, and metric choices complicate comparisons. It presents complementary measures, including accuracy, averaging schemes, Fβ, MCC, ROC curves, and AUC, while noting their distinct limitations.

  • Evaluation Challenges: Comparable evaluation is hindered by nonstandard data collection protocols and different training and test sets, even when datasets share a common source.Different performance measures can also convey different aspects of classification quality, so their meanings must be considered when comparing experiments.
  • Averaging Metrics: Macro-averaging weights categories equally, whereas micro-averaging weights documents equally through pooled contingency-table decisions.The two averaging schemes therefore summarize classifier performance from different perspectives.
  • Aggregated Metrics: Fβ balances recall and precision through β, with F1 assigning them equal weights.Fβ is an aggregated classifier-evaluation metric based on recall and precision.
  • Aggregated Metrics: MCC captures all entries of a confusion matrix and remains a balanced measure for binary classification with uneven class sizes.The paper presents MCC as a measure of binary-classification quality that can accommodate class imbalance.
  • Metric Limitations: No single metric captures every classifier strength and weakness, because one classifier may score higher on MCC while another scores higher on F1.The paper therefore treats metric choice as consequential when comparing classifiers.
  • ROC and AUC: Class imbalance can cause ROC curves to poorly represent classifier performance, despite ROC and AUC being useful evaluation tools.ROC curves plot true-positive rate against false-positive rate, while AUC summarizes the area beneath the curve.

6. Discussion

The discussion compares text-classification representations, dimensionality-reduction methods, classifiers, and evaluation measures while emphasizing their distinct trade-offs and limitations.

  • 6.1. Text and Document Feature Extraction: Weighted-word features count document terms, whereas word embeddings learn vectors from word occurrence and co-occurrence information.Weighted words provide a simple scoring scheme; embeddings incorporate contextual information but require substantial training data.
  • 6.1. Text and Document Feature Extraction: Word embeddings address missing semantic similarity between distinct words, but contextualized representations are needed when a word’s meaning varies by sentence.The paper identifies slang, abbreviations, and polysemy as limitations for conventional embeddings.
  • 6.2. Dimensionality Reduction: Dimensionality reduction primarily targets computational time and memory complexity, but methods differ in data and interpretability requirements.PCA has computational complexity; LDA requires labeled data and a manually chosen component count; random projection performs poorly on small data, while autoencoders need more training data.
  • 6.3. Existing Classification Techniques: The survey compares classifier advantages and limitations, including retrieval restrictions for Rocchio, complexity and interpretability costs for boosting and bagging, and assumptions in logistic regression and Naïve Bayes.KNN is easy to implement and supports multiclass cases but is constrained by storage for large search problems.
  • 6.3.2. State-of-the-Art Techniques’ Comparison: State-of-the-art techniques are compared using architecture, novelty, feature extraction, corpus, validation measure, and limitations.The comparison treats the classifier and feature extraction technique as linked components of each system.
  • 6.4. Evaluation: Accuracy is unsuitable for unbalanced data, while precision and recall are widely used to measure classifier effectiveness.The survey also discusses Fβ Score, MCC, ROC, and area under the ROC curve as evaluation methods.

7. Text Classification Usage

Text classification is used across information management, recommendation, summarization, healthcare, behavioral research, marketing, and law. The surveyed applications show how classification organizes or interprets large volumes of textual information.

  • Information Retrieval: Information retrieval uses text classification to find documents meeting information needs within large collections.Methods mentioned include Naïve Bayes, SVM, decision trees, J48, KNN, and IBK.
  • Information Filtering: Information filtering selects relevant information or rejects irrelevant incoming data to model users’ long-term interests.Bayesian inference networks are commonly used to propagate values and return highly relevant documents.
  • Sentiment Analysis: Sentiment classification assigns opinion-bearing documents to positive or negative categories using lexical, syntactic, and frequency-based features.Naïve Bayes and SVM are identified as popular supervised methods for sentiment classification.
  • Recommender Systems: Content-based recommender systems suggest items from item descriptions and user-interest profiles, which may combine extracted text attributes with directly specified conditions.User profiles can be learned from feedback, search history, self-reports, filters, or query conditions.
  • Document Summarization: Text classification supports document summarization by extracting important document features, including in multi-document settings driven by growing online information.The passage notes that summaries may use words or phrases absent from the original document.
  • Other Domains: Applications also include medical coding, human-behavior research, social-media marketing, and legal-document categorization.These domains involve narrative medical records, informal language data, customer discovery, and large collections of legal documents.

8. Conclusions

The paper surveys text classification as a machine-learning problem and organizes its treatment around feature extraction, dimensionality reduction, algorithms, evaluation, and applications. It also addresses commonly used representations and classification techniques while discussing their use in academic, commercial, and domain-specific settings.

  • Text classification methods are organized around feature extraction, dimensionality reduction, algorithm comparison, evaluation, and applications.
  • Feature extraction methods covered include TF-IDF, term frequency, and word embeddings such as Word2Vec, contextualized representations, GloVe, and FastText.
  • The survey discusses text and document cleaning as a preprocessing step that can improve application accuracy and robustness.
  • The paper compares common text classification algorithms and discusses their use in applications supporting fields such as law and medicine.
  • Recent techniques and trends in text classification algorithms are included in the survey.
Loading 1904.08067v5…