Source-linked AI summary
Cyberbullying Detection in Social Networks Using Deep Learning Based Models; A Reproducibility Study
Maral Dadvar, Kai Eckert
TL;DR
Cyberbullying detection remains an important, unresolved problem, while much prior work used conventional machine learning and models often focused on one platform. The authors reproduce deep-learning models, extend them to YouTube, and report that the models outperform previous machine-learning results on that dataset.
Problem
Cyberbullying detection remains unresolved, and much prior work used conventional machine-learning models, many adaptable to only one social network.
Source: §1 · p. 2Method
The study reproduces and validates prior deep-learning models on Formspring, Twitter, and Wikipedia, then evaluates them on YouTube and across platforms.
Source: §1 · p. 2, §4 · p. 10Results
Deep-learning models coupled with transfer learning outperformed all previous machine-learning results for cyberbullying detection on the YouTube dataset.
Source: §4 · p. 11Takeaways & Limitations
The study reports that its deep-neural-network models were adaptable and transferable to the new YouTube dataset.
Source: §4 · p. 11Whether integrating other information sources, including users’ profile information, improves the models remains for future investigation.
Source: §4 · p. 11
Abstract
from arXiv · showhide
Cyberbullying is a disturbing online misbehaviour with troubling consequences. It appears in different forms, and in most of the social networks, it is in textual format. Automatic detection of such incidents requires intelligent systems. Most of the existing studies have approached this problem with conventional machine learning models and the majority of the developed models in these studies are adaptable to a single social network at a time. In recent studies, deep learning based models have found their way in the detection of cyberbullying incidents, claiming that they can overcome the limitations of the conventional models, and improve the detection performance. In this paper, we investigate the findings of a recent literature in this regard. We successfully reproduced the findings of this literature and validated their findings using the same datasets, namely Wikipedia, Twitter, and Formspring, used by the authors. Then we expanded our work by applying the developed methods on a new YouTube dataset (~54k posts by ~4k users) and investigated the performance of the models in new social media platforms. We also transferred and evaluated the performance of the models trained on one platform to another platform. Our findings show that the deep learning based models outperform the machine learning models previously applied to the same YouTube dataset. We believe that the deep learning based models can also benefit from integrating other sources of information and looking into the impact of profile information of the users in social networks.
1 Introduction
Cyberbullying is a widespread online risk, and automatic detection remains unresolved. This study reproduces prior deep-learning models and examines their performance and transfer across social-media datasets.
1 Introduction
- Adolescents’ online presence exposes them to cyberbullying, a problem reported among children and teenagers across Europe.The cited statistics include 18% involvement, 20% exposure among 11- to 16-year-olds, and 20%–40% cyber-victimization rates.Source: §1 · pp. 1–2
- Automatic detection and prevention could help address cyberbullying, but the problem remains unresolved despite existing tools and support programs.Source: §1 · p. 2
- Most prior detection studies used conventional machine learning, while recent work applied deep neural networks and reported cross-platform transfer.The prior work cited as stated that its models could be adapted to other datasets.Source: §1 · p. 2
- The authors reproduce and validate on Formspring, Twitter, and Wikipedia, then extend evaluation to YouTube and cross-platform transfer.They aim to assess adaptability and compare deep-learning models with conventional machine-learning results on YouTube.Source: §1 · p. 2
2 Reproduced Experimental Setup
The study reproduces the reference experiments on three manually labeled datasets, using shared preprocessing and evaluation procedures. Oversampling improved performance across datasets, while most reproduced results were within the reference results’ standard deviations.
2 Reproduced Experimental Setup
- The experiments reproduced the reference models on Formspring, Wikipedia, and Twitter using the same models and experimental setup.Source: §2 · p. 3
2.1 Datasets
- The three manually labeled datasets are class-imbalanced, with fewer bullying posts; long posts were truncated at each dataset’s 95th-percentile length.The imbalance led the researchers to oversample the bullying class.Source: §2.1 · p. 3
2.2 Models and Methods
- The models used CNN, LSTM, BLSTM, or BLSTM with attention, with shared embedding, dropout, dense-output, and softmax layers.The neural architecture layer varied by model; dropout rates were 0.25 before and 0.5 after it.Source: §2.2 · p. 4
- Word embeddings were initialized randomly, with GloVe, or with SSWE; reported results used 50-dimensional vectors because dimensions 30–200 showed no significant difference.SSWE incorporates text sentiment, whereas GloVe uses syntactic context.Source: §2.2 · p. 4
2.3 Workflow and Results
- After stopword and punctuation removal, models were evaluated with five-fold cross-validation using precision, recall, and F1; bullying posts were tripled in training data.Source: §2.3 · p. 5
- Oversampling significantly improved performance across all datasets, especially Formspring; most reproduced results were within the reference results’ standard deviations.The reported significance test was Mann–Whitney U, P<0.001.Source: Table 1b · p. 5, §2.3 · p. 6
- On original datasets, random embeddings outperformed GloVe and SSWE; on oversampled datasets, embedding choice showed no significant effect.Most reproduced-versus-reference F1 inconsistencies occurred for CNN on Twitter.Source: Table 2a · p. 6, Table 2b · p. 6, §2.3 · p. 7
- With SSWE embeddings, LSTM was outperformed by the other models, whose performance gaps were small; reproduction differences mainly involved recall.Source: §2.3 · p. 7, Table 3a · p. 7, Table 3b · p. 7
3 Application and Evaluation of the Methods on a New Dataset
The study evaluates reproduced deep neural models on YouTube and tests transfer from Twitter, Formspring, and Wikipedia to YouTube.
3.1 YouTube Dataset
- The YouTube dataset contains about 54,000 comments from 3,858 users, with about 6,500 comments labeled as bullying.Source: §3.1 · p. 8
3.2 Workflow and Results
- Bullying posts were tripled to address YouTube class imbalance; experiments compared random, GloVe, and SSWE embeddings across dimensions 25–200.Reported embedding results averaged across the experimented dimensions.Source: §3.2 · p. 8
- P<0.001, oversampling significantly improved performance across all three word embeddings.Source: Table 4 · p. 9
- With SSWE, BLSTM had the highest F1-score and recall, while attention traded lower recall for slightly higher precision; across embeddings, SSWE led F1 and precision, and random led recall.Source: Table 5 · p. 9, §3.2 · p. 9
3.3 Transfer Learning
- Adding network weights to transferred embeddings did not significantly improve over feature-level transfer, suggesting weights were less essential than learned embeddings.Source: §3.3 · p. 10, Table 6 · p. 10
- Feature-level transfer improved precision and recall across datasets; Wikipedia reached F1-score 0.74.Source: Table 6 · p. 10, §3.3 · p. 10
- Complete transfer yielded YouTube F1-scores of 0.30 from Formspring, 0.23 from Wikipedia, and 0.15 from Twitter; recalls were low but varied.Source: §3.3 · p. 9, Table 6 · p. 10
4 Conclusion and Future Work
The study reproduces the reference DNN work and extends evaluation to YouTube, where DNN models with transfer learning outperform prior ML results. It identifies unclear reproduction details and data imbalance as issues for further assessment, and recommends investigating additional information, including user profiles.
4 Conclusion and Future Work
- The reproduction had some inconsistencies that may reflect details and settings in the reference work that were not clearly stated.The authors report that the source code and materials were mostly well organized and accessible.Source: §4 · p. 10
- DNN models coupled with transfer learning outperformed all previous ML results for cyberbullying detection on the YouTube dataset.The authors also report that their DNN models were adaptable and transferable to the new dataset.Source: §4 · p. 11
- Because bullying posts are often scarce, the effects of oversampling them on model performance still need further assessment.Source: §4 · p. 11
- Future work should investigate whether integrating other information, particularly users’ profile information, improves DNN models.Source: §4 · p. 11