Source-linked AI summary
Towards Benchmark Datasets for Machine Learning Based Website Phishing Detection: An experimental study
Abdelhakim Hannousse, Salima Yahiouche
TL;DR
Existing phishing datasets are unsuitable for replication and experiments with new features, so the paper proposes and tests a reproducible, extensible dataset scheme. Random Forest performs best across feature classes, while seven external-service features reach 94.09% accuracy.
Problem
Existing phishing datasets are largely self-collected from different sources and are not suitable for replication or experiments with new features, despite benchmark datasets being important for fair evaluation.
Source: §1 · p. 2, §7 · p. 18Method
The paper proposes guidelines for reproducible, extensible phishing-detection datasets and builds a sample dataset with 87 features and associated DOM-tree objects.
Source: §1 · p. 2, §7 · p. 18Results
Across feature classes, Random Forest outperformed other classifiers, while seven external-service features alone reached 94.09% accuracy.
Source: §7 · p. 19, §8 · p. 20Takeaways & Limitations
Random Forest systems can effectively predict phishing pages in browsers, while external-service features may introduce network delay.
Source: §7 · p. 19, §8 · p. 20The runtime analysis adopts all instances on the assumption that dataset balance makes this feasible despite differing content sizes.
Source: §5.3 · p. 14
Abstract
from arXiv · showhide
In this paper, we present a general scheme for building reproducible and extensible datasets for website phishing detection. The aim is to (1) enable comparison of systems using different features, (2) overtake the short-lived nature of phishing websites, and (3) keep track of the evolution of phishing tactics. For experimenting the proposed scheme, we start by adopting a refined classification of website phishing features and we systematically select a total of 87 commonly recognized ones, we classify them, and we made them subjects for relevance and runtime analysis. We use the collected set of features to build a dataset in light of the proposed scheme. Thereafter, we use a conceptual replication approach to check the genericity of former findings for the built dataset. Specifically, we evaluate the performance of classifiers on individual classes and on combinations of classes, we investigate different combinations of models, and we explore the effects of filter and wrapper methods on the selection of discriminative features. The results show that Random Forest is the most predictive classifier. Features gathered from external services are found the most discriminative where features extracted from web page contents are found less distinguishing. Besides external service based features, some web page content features are found time consuming and not suitable for runtime detection. The use of hybrid features provided the best accuracy score of 96.61%. By investigating different feature selection methods, filter-based ranking together with incremental removal of less important features improved the performance up to 96.83% better than wrapper methods.
1 Introduction
Phishing websites and tactics are evolving, while machine-learning detection depends on effective classifiers, distinguishing features, and representative datasets. Existing datasets often lack source URLs and cannot support replication or feature comparisons, motivating reproducible benchmarks and experiments on features and models.
1 Introduction
- Machine-learning phishing detection is a classification task whose effectiveness depends on classifiers, distinguishing features, and representative training datasets.Systems examine either webpage content or URLs.Source: §1 · p. 2
- Earlier reviews reported different leading classifiers for content-based and URL-based detection, but their statistical findings were not decisive.Das et al. found SVM most used for content-based detection and Decision Tree for URL-based detection.Source: §1 · p. 2
- Prior studies lack agreement on suitable phishing-detection feature classes: Chiew et al. advocate URL-only features, while Jain et al. use content-only features and exclude time-consuming external features.Features should also resist manipulation, and extraction time matters for real-time detection.Source: §1 · p. 2
- 18–20Missing source URLs and short-lived phishing websites limit replication and new-feature experiments; these issues make dataset quality analysis and system comparisons difficult.Source: §1 · p. 2
- The paper proposes a reproducible, extensible dataset-construction scheme and a refined classification of commonly used phishing features.The built dataset is intended to support system comparisons; the paper also examines classifier performance, runtime suitability, feature selection, and model accuracy.Source: §1 · p. 2
2 Related Work
Prior phishing-detection research reports varied classifiers, feature sets, and feature-selection or ensemble approaches, but reviews relied on statistical summaries. This paper experimentally examines those findings across feature classes and combinations.
2 Related Work
- The paper notes a continuing rise in publications on machine-learning phishing detection, with Figure 1 presenting annual publication counts.The figure's results were retrieved on September 16, 2020.Source: §2 · p. 3, Figure 1 · p. 3
- Prior reviews characterized classifier use and runtime or feature trends statistically; this study aims to test those conclusions experimentally.Reviews found runtime performance often neglected and more features associated with better performance; classifier rankings were also summarized in.Source: §2 · p. 3
- Prior studies reported strong results from varied feature sets and classifiers, while this paper compares individual feature classes and their combinations.Reported examples include up to 98.25% accuracy with Random Forest using URL and TF-IDF features, and 94.79% with Random Forest using features from different classes.Source: §2 · p. 3
- Earlier feature-selection work ranked or reduced features using filters and wrappers, but some evaluations covered only one feature class or selected datasets and classifiers.This study tests ranking filters across all feature classes and experiments wrapper algorithms, including Boruta.Source: §2 · pp. 3–4
- Prior phishing systems combined classifiers through stacking or majority voting; this study experiments both types of model combinations.Examples include a Random Forest-to-neural-network ensemble and majority voting across classifiers.Source: §2 · p. 4
3 Scheme for constructing reproducible and extensible datasets
The proposed guidelines address the lack of reproducible phishing-detection benchmarks and the short-lived nature of phishing websites. They specify how to collect, prepare, preserve, and date dataset material for replication and later feature analysis.
3 Scheme for constructing reproducible and extensible datasets
- Benchmark datasets are lacking, while short-lived phishing pages and datasets without URLs constrain content analysis and replication.The cited imbalance study found declines of 5.9% to 42% in F-1 score for imbalanced datasets.Source: §3 · p. 4
- G1 recommends collecting URLs from multiple sources and using suitable seeds and categories to balance homogeneity with URL diversity.Alexa top domains can seed crawling, but are not directly suitable when subdomains and paths matter.Source: §3 · p. 4
- G2 calls for removing duplicate URLs and limiting URLs from the same domain to obtain more representative samples.URLs from one domain are likely to have similar feature values.Source: §3 · p. 4
- G3 recommends preserving URLs as dataset indices or in a separate file so URL-based features can be reproduced and the dataset extended.Source: §3 · p. 4
- G4 recommends saving page DOM trees, enabling content-feature experiments and dataset extension even after links become dead.Source: §3 · p. 4
- G5 calls for shuffled, balanced datasets, while G6 recommends dating datasets to compare systems over time and track phishing-tactic evolution.Source: §3 · p. 4
4 Dataset preparation
The authors assemble a 11,430-URL dataset and organize 87 commonly recognized phishing features across URL, page-content, and external-service classes.
4 Dataset preparation
- The study collected 11,430 phishing and legitimate URLs for its experiments.Figure 2 depicts the overall construction process.Source: §4 · p. 4
4.1 URL collection
- Legitimate URLs came from Alexa and Yandex, while Phishtank and Openphish supplied phishing URLs; duplicate and dead URLs were removed and at most 12 URLs per domain kept.The Alexa collection included 6,000 URLs crawled from its 500 top domains.Source: §4.1 · p. 5
4.2 Feature extraction
- Review-based deduplication and standardized measures yielded 87 features, classified as common, alternative, or optional; a Python script maps each URL to an indexed 87x1 vector.The refined classification distinguishes internal URL and page-content features from optional external features.Source: §4.2 · pp. 5–6
URL-based features
- URL-based features analyze URL text, using structural properties such as protocol or domain parts and statistical properties such as counts or distributions.Some structural features may require external services for estimation.Source: URL-based features · p. 6
Content-based features
- Page-content features analyze HTML hyperlinks and abnormal content, including suspicious content or scripts such as empty links and form submissions to email addresses.The paper groups these as hyperlink and abnormal-content features.Source: Content-based features · p. 8
External-based features
- External features are obtained by querying third-party services and search engines, including WHOIS, Alexa, Openpagerank, and Google.The paper lists these features in Table 4.Source: External-based features · p. 9, Table 4 · p. 9
4.3 Dataset generation
- Dataset B stores URL-indexed feature vectors, balances legitimate and phishing classes at 50% each, shuffles rows, and is released with extraction code and dataset A for replication.Dataset A contains source URLs and generated DOM trees; dataset B contains the extracted feature vectors.Source: §4.3 · p. 10
5 Experiments
The five experiments evaluate classifiers and feature-class combinations, test filter and wrapper selection methods, and measure feature-extraction time for runtime suitability.
5 Experiments
- The experiments use 10-fold cross-validation and evaluate classifier performance with accuracy and Macro F1-score.Accuracy is the ratio of correct predictions to samples; Macro F1 averages class-wise F1-scores.Source: §5 · p. 10
5.1 Classifier evaluation
- Five classifiers are compared across feature classes, with the best per-class classifiers considered for combinations.The classifiers are Decision Tree, Random Forest, Logistic Regression, Naïve Bayes, and SVM.Source: §5.1 · p. 11, Experiment II · p. 11
Experiment I
- Individual classifiers are evaluated on separate feature classes and hybrid feature-class combinations.The best-performing classifier for each class becomes a candidate for the next experiment.Source: Experiment I · p. 11, Figure 4 · p. 11
Experiment II
- Three per-class classifiers are combined using AND, OR, stacking, and majority voting.Stacking trains a new classifier on the three base models’ outputs; voting selects the prediction receiving more than one vote.Source: Experiment II · pp. 11–12, Figure 5 · p. 12
5.2 Feature selection methods evaluation
- Feature-selection experiments test filters and wrappers for identifying discriminative subsets and improving model accuracy.The study examines the hypothesis that higher-ranked features provide higher classification accuracy.Source: §5.2 · p. 12
Experiment III
- Four ranking filters are applied, then low-ranked features are removed incrementally while classifiers are tested on the remainder.The filters are chi-square, Pearson correlation, information gain, and Relief.Source: Experiment III · p. 12, Figure 6 · p. 13
Experiment IV
- Wrapper methods select feature subsets according to a chosen classifier; the study tests Weka evaluators and Boruta.Best-First search is used with Weka evaluators, while Boruta is also examined as a wrapper algorithm.Source: Experiment IV · pp. 12–13, Figure 7 · p. 13
5.3 Experiment V: Feature extraction runtime analysis
- Feature-extraction times are estimated across all 11,430 dataset samples to identify features suitable for runtime detection.The study examines feature classes first, then inspects individual features when a class shows notable delays; using all samples is feasible because the dataset is balanced.Source: §5.3 · pp. 13–14
6 Experimental results
Across five experiments, the study compares classifiers, feature combinations, selection methods, and extraction times. Chi-square ranking reaches 96.83% accuracy with 73 features, exceeding the reported wrapper-method results.
6.1 Experiment I. Performance of classifiers trained on different classes of features
- Random Forest outperforms the other classifiers, while external-based features achieve 94.09% accuracy, ahead of URL-based at 91.03% and content-based at 89.87%.These are accuracy scores for Random Forest on individual feature classes.Source: §6.1 · p. 14, Figure 8 · p. 14
- 96.61% accuracy is achieved with all features, while URL and external features together reach 96.60%.Both results use Random Forest; the all-feature combination has a 96.60% Macro F1-score.Source: §6.1 · pp. 14–15, Figure 9 · p. 15
6.2 Experiment II. Performance of combined models each trained on a different class of features
- Combining three per-class Random Forest models with AND, OR, stacking, or majority voting does not improve accuracy.Stacking and majority voting score approximately as well as a single model trained on content and external features.Source: §6.2 · p. 15, Figure 10 · p. 15
6.3 Experiment III. Performance by training on features selected using filter ranking methods
- The seven features shared by all four filters reach 94.67% accuracy with Random Forest, while top-ranked features span all three feature classes.google_index is ranked first by every filter; external-service features are found important, while content features are less important.Source: §6.3 · p. 16
- 96.83% accuracy is reached with 73 features using chi-square ranking, and adding lower-ranked features slightly reduces accuracy.The result is for Random Forest; wrapper evaluators do not improve its accuracy over decremental filter-based selection.Source: §6.3 · p. 16, §6.4 · p. 17
6.4 Experiment IV. Performance by training on features selected using wrapper methods
- ClassifierSubsetEval fails to improve Random Forest accuracy; WrapperSubsetEval and Boruta perform similarly, but Boruta takes about 122 seconds versus about 24 hours.All three wrapper evaluators fail to improve accuracy over decremental selection based on filter ranking.Source: §6.4 · p. 17, Table 7 · p. 17
6.5 Experiment V. Runtime analysis of feature extraction
- All URL-based features take about 41.5 ms to extract; external-service features take about 5 seconds, reducible to about 4 seconds by combining WHOIS calls.Hyperlink features f63 and f65 account for delays in content extraction because they check every page link.Source: §6.5 · p. 18, Figure 12 · p. 18
7 Discussion and lessons learned
The paper proposes reproducible, extensible phishing-detection benchmarks and evaluates feature classes, classifiers, and selection methods. Its findings highlight strong external-service features, runtime trade-offs, and accuracy gains from feature ranking with incremental removal.
7 Discussion and lessons learned
- The proposed six-guideline process produced a dataset of 87 features and a separate DOM-object dataset intended to support replication, personalized feature testing, and fair comparison.The datasets and Python scripts are publicly available.Source: §7 · p. 18
- Feature importance can shift across datasets and over time: none of 5,715 phishing pages used stolen-information emails, and only 97 used IP addresses.The paper proposes repeated testing on benchmark datasets to track feature significance; HTTPS use may also reduce f25's future value.Source: §7 · pp. 18–19
- Hyperlink features f63 and f65 were slower than all external-service features, making both unsuitable for real-time detection; only f65 was rejected by WrapperSubsetEval.The other examined feature-selection methods did not identify these features as less important.Source: §7 · p. 19
- URL and external-service models reached 96.65% accuracy, while the examined OR combination reached 89.18%; URL feature extraction took milliseconds, with about four seconds sufficient for its feature vector.Additional time is still needed for the model to classify the instance.Source: §7 · p. 19
- External-service features achieved 94.09% accuracy using only seven features; Phishtank presence was excluded because Phishtank supplied phishing URLs for the dataset.Adding Phishtank presence may improve performance but would add network delay.Source: §7 · p. 19
- Chi-square ranking improved model accuracy by up to 0.22%, and incremental removal improved it by up to 0.25%, outperforming wrapper evaluators except Boruta on time.Classifiers, including Random Forest, were sensitive to feature order.Source: §7 · p. 19, Table 8 · p. 19
- A passive Chrome plugin uses the study's best model, a Random Forest trained on 73 chi-square-ranked features, to report page legitimacy.It can also be used to compare its efficiency with Chrome's active warning.Source: §7 · pp. 19–20
8 Conclusion
The conclusion presents reproducible, extensible datasets as benchmarks for phishing-detection systems and summarizes the study's classifier, feature, runtime, and selection findings. It also identifies further dataset validation and deep-learning comparisons as future work.
8 Conclusion
- Random Forest systems had comparatively higher accuracy than other classifiers across the examined feature classes, while hybrid features outperformed single-class features.Source: §8 · p. 20
- External-service features were more discriminative despite their small number, while examined content features were less discriminative and some hyperlink features could cause severe network delays.External-service features may also cause network delay.Source: §8 · p. 20
- Filter ranking with incremental removal of less important features yielded better accuracy than wrapper methods and reduced the feature set.The conclusion notes that Random Forest is sensitive to feature order.Source: §8 · p. 20
- The proposed guidelines are intended to support reproducible, extensible phishing-detection datasets that can serve as machine-learning benchmarks.Source: §8 · p. 20
- Future work will validate the conclusions on more datasets built using the guidelines and compare deep-learning approaches.Source: §8 · p. 20