Source-linked AI summary
Naive Bayes and Text Classification I - Introduction and Theory
Sebastian Raschka
TL;DR
This article introduces naive Bayes classification for document categorization, covering its probabilistic foundations, assumptions, decision rule, and text-processing choices. It explains how priors, smoothing, tokenization, and model selection affect classification, while noting that performance depends on data, features, and independence assumptions.
Problem
The article addresses how models can learn from available data to make decisions and predictions, focusing on document categorization with naive Bayes classifiers.
Method
The article explains naive Bayes using Bayes' theorem, conditional-independence assumptions, posterior decision rules, and text-processing methods such as tokenization, stemming, and lemmatization.
Results
Naive Bayes classifiers are described as robust, easy to implement, fast, and accurate, while the multinomial model tends to outperform the multivariate Bernoulli model for relatively large vocabularies.
Takeaways & Limitations
Representative training data, feature choices, and comparisons between classification models are important when applying naive Bayes to text classification.
Takeaways & Limitations
Strong violations of feature-independence assumptions and non-linear classification problems can lead to very poor naive Bayes performance.
Abstract
from arXiv · showhide
Naive Bayes classifiers, a family of classifiers that are based on the popular Bayes' probability theorem, are known for creating simple yet well performing models, especially in the fields of document classification and disease prediction. In this article, we will look at the main concepts of naive Bayes classification in the context of document categorization.
1 Introduction
Pattern classification trains models on available data to make decisions and predictions automatically. The article introduces naive Bayes as a simple, effective classifier, especially for document categorization.
- Pattern classification and machine learning seek models that learn from available data to make decisions and predictions.
- Applications of learned models include search engines, text recognition, barcode scanning, disease diagnosis, and speech recognition.
- Supervised pattern classification trains models on labeled data to assign predefined class labels to new objects.
- The article uses spam filtering as an example of classifying new text messages as spam or not-spam with naive Bayes.
- Naive Bayes classifiers are based on Bayes’ probability theorem and are known for simple yet well-performing models in document classification and disease prediction.
2.1 Overview
Naive Bayes classifiers are fast, simple linear probabilistic models that can work well despite their independence assumption. Their suitability depends on the data and problem structure, so alternatives should be compared on the target dataset.
- Naive Bayes classifiers use Bayes’ theorem and assume that dataset features are mutually independent.The independence assumption is often violated, yet the classifiers can still perform well.
- Naive Bayes classifiers are relatively robust, easy to implement, fast, and accurate across applications including disease diagnosis, RNA classification, and spam filtering.
- Strong independence violations and non-linear problems can produce very poor naive Bayes performance, making model choice dependent on the data and task.
- Comparing classifiers on the particular dataset should consider both prediction performance and computational efficiency.
- The article applies naive Bayes to a toy problem and then trains a classifier on a public SMS collection to classify unseen messages as spam or ham.
2.2 Posterior Probabilities
Naive Bayes uses Bayes’ theorem to estimate how likely an object is to belong to each class given its observed features. Classification selects the class with the highest posterior probability, combining likelihoods with prior probabilities.
- Posterior probability equals conditional probability multiplied by prior probability, while the evidence denominator can be omitted when comparing classes.
- Bayes’ theorem provides the foundation for naive Bayes classification and defines posterior probability for class membership given observed features.
- Classifiers approximate decision boundaries between classes, but naive Bayes is unsuitable when classes form non-linearly separable problems.
- In the notation, x_i denotes a sample’s feature vector, ω_j denotes a class, and P(x_i | ω_j) denotes observing x_i given class ω_j.
- The classification objective is to maximize the posterior probability given the training data and use it to formulate the decision rule.
2.3 Class-conditional Probabilities
Naive Bayes estimates class-conditional probabilities from training data by assuming conditional independence among features. This turns a joint likelihood into a product of feature likelihoods, typically estimated from observed frequencies.
- Naive Bayes assumes conditional independence of features so class-conditional probabilities can be estimated directly from training data.
- For a d-dimensional feature vector, the class-conditional probability is calculated as the product of the likelihoods for all d features.
- Individual feature likelihoods can be estimated by maximum likelihood, which is a frequency for categorical data.
- In the spam example, the likelihood of “Hello World” given spam is the product of the likelihoods for “hello” and “world.”
- The independence assumption is violated when words influence one another, as with “peanut” and “butter,” although naive Bayes can still perform well.
2.4 Prior Probabilities
Prior probabilities encode how likely each class is before observing features and influence naive Bayes decision regions. With equal priors, class-conditional probabilities determine the boundary; increasing one class prior expands its decision region.
- Prior probabilities can be introduced as prior belief or estimated from representative training data using class frequencies.
- Class priors represent the general probability of encountering a particular class, such as the probability that a new message is spam.
- With uniform priors, the decision boundary depends on the class-conditional probabilities and lies between the two distributions.
- Increasing the prior probability of class ω1 expands its decision region by moving the boundary toward the other class.
- When feature evidence is equally likely across classes, the decision can depend entirely on prior knowledge, such as classifying a message as ham when nine of ten messages are assumed non-spam.
2.5 Evidence
The evidence term is the probability of observing a feature pattern independently of its class label. Although needed for exact posterior probabilities, it can be omitted from the decision rule because it scales competing posteriors equally.
- Evidence P(x) denotes the probability of encountering pattern x independently of the class label.
- The evidence term is required to calculate posterior probabilities accurately but can be removed when comparing class posteriors in the decision rule.
- Because evidence acts as a scaling factor shared by the competing classes, it does not change which posterior probability is larger.
2.6 Multinomial Naive Bayes - A Toy Example
The toy example classifies samples using color and shape features, class priors, and class-conditional probabilities under the feature-independence assumption. It also shows why unseen feature values require additive smoothing to avoid zero posterior probabilities.
- 2.6 Multinomial Naive Bayes - A Toy Example: The toy dataset contains 12 samples from two classes, with each sample described by color and geometrical shape.
- 2.6 Multinomial Naive Bayes - A Toy Example: Naive Bayes estimates the prior probabilities from class frequencies when samples are assumed to be independently and identically distributed.
- 2.6 Multinomial Naive Bayes - A Toy Example: Under the feature-independence assumption, the class-conditional probability is computed as the product of the individual conditional probabilities for color and shape.
- 2.6.2 Classification: For x = [blue, square], the posterior values are 0.18 for class + and 0.15 for class −, so the sample is classified as +.
- 2.6.3 Additive Smoothing: An unseen color such as yellow produces a zero class-conditional probability and therefore a zero posterior under the unsmoothed model.
- 2.6.3 Additive Smoothing: Lidstone smoothing uses α < 1 and Laplace smoothing uses α = 1 to avoid zero probabilities in the multinomial model.
3 Naive Bayes and Text Classification
Naive Bayes text classification represents documents as features, preprocesses those features, and applies posterior-probability decision rules under conditional independence assumptions.
- Text classification trains a model on labeled data to assign predefined class labels to new objects, including spam and ham messages.
- Good features should be salient, invariant, and discriminatory for the classification problem.
- 3.1 The Bag of Words Model: The bag-of-words model builds a vocabulary of distinct training-set words while ignoring word order, then represents documents as d-dimensional vectors.The vector dimension equals the vocabulary size, d = |V|; this conversion is called vectorization.
- The feature representation determines whether values are binary presence indicators or absolute term-frequency counts, depending on the Bernoulli or Multinomial model.
- 3.1.1 Tokenization: Tokenization splits text into individual elements and may remove punctuation and stop words, lowercase letters, stem or lemmatize words, and construct n-grams.An n-gram is a sequence of n items, and the optimal n depends on the language and application.
- 3.2 The Decision Rule for Spam Classification: Naive Bayes classifies a message as spam when its spam posterior is at least its ham posterior, with the posterior proportional to likelihood times prior.The class priors can be estimated from spam and ham frequencies in the training data, and the evidence denominator can be omitted because it is constant across classes.
3.3 Multi-variate Bernoulli Naive Bayes
Bernoulli and multinomial naive Bayes use different text representations and probability estimates, while feature-engineering choices substantially affect classification performance.
- 3.3 Multi-variate Bernoulli Naive Bayes: The Multivariate Bernoulli model represents each vocabulary token with a binary value indicating whether it occurs in a document.Its feature vector has one dimension for each vocabulary word.
- Bernoulli likelihoods estimate the probability that each token occurs in a class, with Laplace-smoothing parameters included in the estimates.
- 3.4 Multinomial Naive Bayes: The multinomial model instead uses term frequency, counting how often each word appears and optionally normalizing counts by document length.
- Multinomial class-conditional probabilities are estimated from class-specific token-frequency totals with vocabulary-size smoothing.
- Under conditional independence, the likelihood of a document is the product of the individual word likelihoods.
- Tf-idf weights terms by frequency while reducing the influence of words appearing across many documents, and it can also be used with naive Bayes classification.
- 3.4.3 Performances of the Multi-variate Bernoulli and Multinomial Model: The multinomial model tends to outperform the multivariate Bernoulli model for relatively large vocabularies, but feature choices can create large performance differences.Recommended comparisons vary the model together with stop-word removal, stemming, and token-length choices.
4 Variants of the Naive Bayes Model
Naive Bayes extends beyond categorical text models to Gaussian modeling of continuous features, while its eager-learning design enables fast prediction after training.
- 4.1 Gaussian Naive Bayes: Gaussian naive Bayes models continuous features by assuming their probability distributions are normal and estimating the sample mean and standard deviation from training data.
- Under conditional independence, the Gaussian class-conditional probability is computed as the product of individual feature probabilities.
- Naive Bayes is an eager learner that builds a model when training data becomes available, making classification of new instances relatively fast.The computationally expensive step is model building rather than classifying new instances.
- Lazy learners train quickly but typically predict more slowly because they re-evaluate and retain the training data.