Source-linked AI summary
Convolutional Neural Networks for Toxic Comment Classification
Spiros V. Georgakopoulos, Sotiris K. Tasoulis, Aristidis G. Vrahatis, Vassilis P. Plagianakos
TL;DR
Online toxic comments can cause harassment and bullying, while existing detection models remain insufficiently reliable and often do not determine toxicity degree. This paper applies CNNs with word embeddings to Kaggle Wikipedia talk-page comments and compares them with Bag-of-Words-based methods. CNNs outperform the traditional approaches, providing evidence for their use in toxic-comment classification.
Problem
Online toxicity creates harassment and bullying risks, while existing detection models are not fully reliable and usually do not determine toxicity degree.
Method
The paper evaluates CNN-based text classification with word embeddings against traditional Bag-of-Words representations and established classifiers on a Kaggle toxic-comment dataset.
Results
CNN models outperform SVM, kNN, NB, and LDA across reported metrics, with accuracy almost over 90% and precision above 90%.
Takeaways & Limitations
The findings provide evidence that CNNs are appropriate for toxic-comment classification and motivate further development of CNN-based text-mining methods.
Abstract
from arXiv · showhide
Flood of information is produced in a daily basis through the global Internet usage arising from the on-line interactive communications among users. While this situation contributes significantly to the quality of human life, unfortunately it involves enormous dangers, since on-line texts with high toxicity can cause personal attacks, on-line harassment and bullying behaviors. This has triggered both industrial and research community in the last few years while there are several tries to identify an efficient model for on-line toxic comment prediction. However, these steps are still in their infancy and new approaches and frameworks are required. On parallel, the data explosion that appears constantly, makes the construction of new machine learning computational tools for managing this information, an imperative need. Thankfully advances in hardware, cloud computing and big data management allow the development of Deep Learning approaches appearing very promising performance so far. For text classification in particular the use of Convolutional Neural Networks (CNN) have recently been proposed approaching text analytics in a modern manner emphasizing in the structure of words in a document. In this work, we employ this approach to discover toxic comments in a large pool of documents provided by a current Kaggle's competition regarding Wikipedia's talk page edits. To justify this decision we choose to compare CNNs against the traditional bag-of-words approach for text analysis combined with a selection of algorithms proven to be very effective in text classification. The reported results provide enough evidence that CNN enhance toxic comment classification reinforcing research interest towards this direction.
1 INTRODUCTION
Online communication generates large volumes of short text, but sparse representations make similarity and classification difficult. The paper examines CNNs for toxic-comment detection and compares them with established text-mining approaches.
- Short-text collections produce sparse matrices with little shared context, making efficient similarity measures difficult for word-frequency approaches.
- CNN-based text-classification research has explored distributed and discrete word embeddings without requiring syntactic or semantic language knowledge.
- Online toxicity includes rude, disrespectful, personal-attack, harassment, and bullying behaviors, creating risks for participation and online safety.The Wikimedia Foundation found that 54% of people experiencing online harassment reported decreased participation in the affected project.
- Real-time toxic-comment identification could help prevent adverse effects, while machine-learning services such as Perspective automatically score abusive speech.
- Existing models are described as insufficiently reliable and generally unable to determine toxicity degree.
- The study investigates whether CNNs using word embeddings improve on traditional Bag-of-Words representations and established classification algorithms.
2 CONVOLUTIONAL NEURAL NETWORKS
The paper adapts CNNs to text by encoding words as dense vectors and processing them through convolutional, pooling, and fully connected layers. Training uses backpropagation to minimize classification error.
- CNN architecture: Convolutional layers apply shared kernel matrices to inputs and produce feature matrices with added biases.Kernel weights and biases are trained as shared neuron connection weights.
- CNN architecture: Pooling layers reduce dimensionality by subsampling neighboring convolutional features, commonly through max-pooling.Max-pooling selects the maximum value from each local neighborhood.
- CNN architecture: Fully connected layers act as trainable feed-forward hidden layers used in the final stages of CNNs.They can be interpreted as convolutional layers with kernel size 1 × 1.
- Training: Backpropagation compares CNN outputs with target vectors, while mini-batch stochastic gradient descent minimizes the cost function.The training error is the difference between each output and its corresponding desirable target.
- Text representation: CNN text classification encodes each word as a dense fixed-size vector before applying the standard CNN pipeline.A vocabulary indexes words, encoded documents become matrices, and the embedding layer transforms rows into low-dimensional representations.
3 BAG-OF-WORDS FOR TEXT MINING
The Bag-of-Words approach represents documents through word-occurrence features while discarding word order and structure. For toxic-comment data, the resulting document-term matrix is visualized with PCA and t-SNE, whose class-colored projections show cluster structures.
- Bag-of-Words representation: Bag-of-Words represents each document using known vocabulary words and their presence or frequency, without retaining word order or document structure.Despite this simplification, BoW models commonly perform well in text categorization and classification.
- Document-term matrix: The toxic-comment document-term matrix is built from case-insensitive unique words after removing punctuation, numbers, and stop words.The procedure then uses term frequency–inverse document frequency rather than raw document word counts.
- Visualization: PCA and t-SNE project the document-term matrix into two dimensions for visual validation, with colors marking samples from different classes.The projections use samples selected according to the experimental-analysis procedure.
- Visualization: Cluster structures appear in both representations, indicating appropriate circumstances for applying a learning algorithm to the resulting matrix.Classification results for methodologies applied to the matrix are reported in the experimental-results section.
4 EXPERIMENTAL ANALYSIS
The evaluation compares CNNs with four traditional classifiers on balanced binary toxic-comment data, using word representations and bag-of-words features. CNN variants achieve stronger and more consistent diagnostic performance across the reported experiments.
- Experimental setup: The study compares word-embedding CNNs with BoW-based SVM, Naive Bayes, kNN, and LDA classifiers for toxic-comment detection.The CNN variants are CNNrand, with randomly initialized trainable word representations, and CNNfix, with fixed pretrained word2vec representations.
- Experimental setup: The evaluation uses human-labeled Wikipedia talk-page comments converted to binary toxic/non-toxic classification.All six original toxicity categories are treated as toxic, and non-toxic comments are randomly subsampled to create balanced datasets.
- Experimental setup: The CNN architecture applies three convolutional layers with filter heights 3, 4, and 5, followed by max-over-time pooling and a fully connected layer.The filters have width 128 and operate on 300-dimensional word vectors.
- Results: CNNrand and CNNfix outperform SVM, kNN, NB, and LDA across the reported classification measures, with accuracy almost over 90%.The comparison includes accuracy, false discovery rate, specificity, precision, recall, and F1-score.
- Results: CNN models achieve precision above 90%, while the other methods range from 65 to 85 percent.CNNfix has the best performance with respect to precision and recall, while CNN models also show the lowest variance and false discovery ratio.
- Results: Naive Bayes and kNN have the weakest results, with the lowest precision and recall respectively.The reported error patterns indicate that kNN labels several non-toxic comments as toxic, whereas NB labels several toxic comments as non-toxic.
5 CONCLUSIONS
The conclusion reports that CNNs can outperform established text-mining methods for toxic-comment classification. It presents these results as support for further CNN-based text-mining development and additional comparisons.
- 5 CONCLUSIONS: CNNs can outperform well-established text-mining methodologies for toxic-comment classification.The authors state that the results provide evidence supporting CNN use for this task.
- 5 CONCLUSIONS: The authors identify adaptive learning methods and n-gram-based comparisons as directions for future CNN-based text-mining research.These directions are motivated by the reported promising results.