Source-linked AI summary
On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis
Jose Camacho-Collados, Mohammad Taher Pilehvar
TL;DR
The paper asks how overlooked preprocessing choices affect neural text classification and evaluates tokenization, lowercasing, lemmatization, and multiword grouping with standard neural models across categorization and polarity tasks. The experiments find that simple tokenization is generally adequate, but preprocessing choices can produce substantial performance variation and matter in particular settings.
Problem
Extrinsic evidence was limited on how preprocessing embedding-training corpora affects the performance of neural text classifiers, while the effects of small input-text decisions on CNNs were also uncertain.
Method
The paper evaluates four preprocessing choices with standard word-based CNN and CNN+LSTM classifiers on topic categorization and polarity detection benchmarks.
Results
Simple tokenization is generally adequate, performing in the same ballpark as the best result in 8 of 9 datasets for both models, while preprocessing variability averages ±2.4% for the best CNN+LSTM model.
Takeaways & Limitations
Preprocessing choices should be selected and reported carefully when training or comparing neural text-classification systems, particularly when training data is limited.
Takeaways & Limitations
The analysis does not cover other factors such as corpus nature, domain, or hyperparameter variation, and further experimentation is needed to fully assess the findings.
Abstract
from arXiv · showhide
Text preprocessing is often the first step in the pipeline of a Natural Language Processing (NLP) system, with potential impact in its final performance. Despite its importance, text preprocessing has not received much attention in the deep learning literature. In this paper we investigate the impact of simple text preprocessing decisions (particularly tokenizing, lemmatizing, lowercasing and multiword grouping) on the performance of a standard neural text classifier. We perform an extensive evaluation on standard benchmarks from text categorization and sentiment analysis. While our experiments show that a simple tokenization of input text is generally adequate, they also highlight significant degrees of variability across preprocessing techniques. This reveals the importance of paying attention to this usually-overlooked step in the pipeline, particularly when comparing different models. Finally, our evaluation provides insights into the best preprocessing practices for training word embeddings.
1 Introduction
The paper examines whether small preprocessing choices affect word-based neural text classifiers, addressing limited evidence about their extrinsic impact. It evaluates CNN-based models on topic categorization and polarity detection and finds that these choices matter in some cases.
- Prior neural-model research paid limited attention to tokenization, lemmatization, lowercasing, and multiword grouping, despite earlier study in conventional classification.
- Embedding-corpus preprocessing had remained understudied in extrinsic neural-network performance, although corpus nature and domain may also matter.
- The paper focuses on how input text is split into meaning-bearing tokens and how preprocessing affects standard CNN-based neural text classifiers.
- The evaluation covers topic categorization and polarity detection, the two tasks used to study preprocessing effects.
- The study asks whether seemingly small input-text preprocessing decisions affect neural network architectures, particularly CNNs.
2 Text Preprocessing
The paper treats tokenization as the baseline and examines lowercasing, lemmatization, and multiword grouping as alternative preprocessing operations. These techniques respectively alter token form, normalize inflection, or combine consecutive tokens into single units.
- The vanilla representation gathers words through tokenization as the input units for classification models.In the running example, whitespace-delimited words become separate tokens.
- The study compares tokenization with three additional techniques: lowercasing, lemmatizing, and multiword grouping.
- 2.1 Lowercasing: Lowercasing converts every input token to lowercase, reducing sparsity and vocabulary size but potentially increasing ambiguity.The paper illustrates this trade-off with Apple the company versus apple the fruit.
- Lemmatization replaces each token with its corresponding lemma to reduce sparsity across inflected forms, potentially neglecting syntactic nuances.The running example changes forms such as asking and manufacturers to ask and manufacturer.
- Multiword grouping combines consecutive tokens found in an inventory into one token because expressions such as United States may have meanings not traceable from individual words.The paper motivates grouping as a way to represent idiosyncratic multiword expressions as single units.
3 Evaluation
The evaluation compares preprocessing choices across topic categorization and polarity detection using CNN and CNN+LSTM classifiers. Results show substantial variability, with simple tokenization generally competitive but multiword-aware embeddings often advantageous in cross-preprocessing tests.
- Experimental setup: The evaluation covers topic categorization and polarity detection under matched and cross-preprocessed settings.It uses CNN and CNN+LSTM classifiers across multiple benchmark datasets.
- Experimental setup: Four preprocessing techniques were applied to both evaluation datasets and the UMBC corpus used to train word embeddings.The techniques were tokenization, lowercasing, lemmatization, and multiword grouping.
- Experiment 1: Preprocessing effect: Preprocessing results varied by dataset, with average variability of ±2.4% for CNN+LSTM and statistically significant differences in seven of nine datasets.Accuracy was averaged across two runs, and significance was assessed with an unpaired t-test at the 5% level.
- Experiment 1: Preprocessing effect: Tokenization alone was near the best result in 8 of 9 datasets for both models, but performed poorly on the specialized Ohsumed medical dataset.Lowercasing and lemmatization exceeded vanilla tokenization by over six points on Ohsumed, where domain-specific meaning and sparsity are concerns.
- Experiment 1: Preprocessing effect: Lemmatization and multiword grouping generally did not improve matched preprocessing performance, because neural architectures can overcome sparsity through word-embedding generalization.This contrasts with their reported usefulness in some conventional linear models.
- Experiment 2: Cross-preprocessing: Multiword-enhanced embeddings performed best on vanilla-tokenized datasets for the single CNN in seven of nine datasets and remained competitive for CNN+LSTM.Using multiword-wise embeddings with vanilla inputs also outperformed the corresponding multiword-grouped setting in eight of nine datasets.
4 Conclusions
The evaluation shows that preprocessing choices materially affect standard word-based neural text classification, although simple tokenization is generally sufficient. Results also identify important exceptions, variability, and implications for comparing systems and training word embeddings.
- Simple tokenization generally performs equally well or better than lemmatization and multiword grouping.
- These findings support careful preprocessing selection and consistent reporting when evaluating or comparing neural systems.
- Domain-specific datasets, including the medical dataset, are an exception where tokenization alone performs poorly.
- Word embeddings trained on multiword-grouped corpora perform well when applied to simply tokenized datasets.The authors suggest this may partially explain the success of pretrained Word2vec embeddings.
- Preprocessing choices produce high result variance, averaging ±2.4% for the best-performing model, especially with insufficient training data.