PaperlayerView exact revision on arXiv

Source-linked AI summary

URLTran: Improving Phishing URL Detection Using Transformers

Pranav Maneriker, Jack W. Stokes, Edir Garcia Lazo, Diana Carutasu, Farid Tajaddodianfar, Arun Gururajan

arXiv:2106.05256v3cs.CR

TL;DR

Phishing URL detection must address rare, short-lived attacks and a growing set of known phishing sites. The paper evaluates transformer-based URLTran approaches and reports higher detection at very low false positive rates, including under specified adversarial scenarios.

  • Problem

    Phishing URLs are rare, campaigns can move multiple times daily, and the growing number of known phishing sites makes list-only blocking inadequate.

    Source: §1 · p. 1

  • Method

    The paper compares URL-only transformer pre-training and fine-tuning with fine-tuning pretrained BERT and RoBERTa, and evaluates adversarially augmented training.

    Source: §1 · p. 2

  • Results

    At 0.01% FPR, URLTran achieves an 86.80% TPR versus 71.20% for URLNet; transformers outperform recent baselines across a wide range of low FPRs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Adversarially augmented training can make transformers robust to novel attacks under specific threat models.

    Source: §6 · p. 8

    The benign labels are limited to URLs validated by analysts or production-grade offline automated analysis, rather than simply unknown URLs.

    Source: §2 · p. 2

Abstract

from arXiv · show

Browsers often include security features to detect phishing web pages. In the past, some browsers evaluated an unknown URL for inclusion in a list of known phishing pages. However, as the number of URLs and known phishing pages continued to increase at a rapid pace, browsers started to include one or more machine learning classifiers as part of their security services that aim to better protect end users from harm. While additional information could be used, browsers typically evaluate every unknown URL using some classifier in order to quickly detect these phishing pages. Early phishing detection used standard machine learning classifiers, but recent research has instead proposed the use of deep learning models for the phishing URL detection task. Concurrently, text embedding research using transformers has led to state-of-the-art results in many natural language processing tasks. In this work, we perform a comprehensive analysis of transformer models on the phishing URL detection task. We consider standard masked language model and additional domain-specific pre-training tasks, and compare these models to fine-tuned BERT and RoBERTa models. Combining the insights from these experiments, we propose URLTran which uses transformers to significantly improve the performance of phishing URL detection over a wide range of very low false positive rates (FPRs) compared to other deep learning-based methods. For example, URLTran yields a true positive rate (TPR) of 86.80% compared to 71.20% for the next best baseline at an FPR of 0.01%, resulting in a relative improvement of over 21.9%. Further, we consider some classical adversarial black-box phishing attacks such as those based on homoglyphs and compound word splits to improve the robustness of URLTran. We consider additional fine tuning with these adversarial samples and demonstrate that URLTran can maintain low FPRs under these scenarios.

1 INTRODUCTION

Phishing detection faces severe class imbalance, short-lived campaigns, and a growing number of known sites, motivating URL-based machine learning. The paper evaluates transformer approaches and reports improved detection at very low false positive rates.

1 INTRODUCTION

  • Phishing detection is difficult because phishing pages are rare, campaigns can move multiple times daily, and known phishing sites keep increasing.The authors argue that blocking only a continuously growing list of known sites often fails to protect users in practice.Source: §1 · p. 1
  • Transformers have advanced NLP, and their use on structured inputs such as tables, code, and SQL motivates applying them to URLs.The paper notes that URL syntax is stricter than natural language syntax.Source: §1 · p. 2
  • URLTran compares URL-only transformer pre-training and fine-tuning with fine-tuning publicly available BERT and RoBERTa models.The study also simulates homoglyph substitutions, compound-word splits, and reordered parameters to examine adversarial robustness.Source: §1 · p. 2
  • At 0.01% FPR, URLTran achieves an 86.80% TPR versus 71.20% for URLNet, a 21.9% relative increase.The authors report that transformers outperform recent state-of-the-art phishing URL models over a wide range of low FPRs.Source: §1 · p. 2
  • The contribution includes analyzing hyperparameters, pre-training tasks, and tokenizers, as well as examining adversarially generated URLs to understand URLTran’s limitations.The work presents URLTran as a large-scale system using production data and labels.Source: §1 · p. 2

2 PHISHING URL DATA

The datasets comprise labeled URLs collected from Microsoft production browsing telemetry over six weeks, with separate training, validation, and test periods. The training and validation sets were down-sampled to address the imbalance between phishing and benign URLs.

2 PHISHING URL DATA

  • The datasets came from Microsoft Edge and Internet Explorer production browsing telemetry collected during summer 2019.Each dataset contains a browsing URL and a boolean phishing-or-benign label.Source: §2 · p. 2
  • Six weeks of historical data were split into four weeks for training, one for validation, and one for testing.The training, validation, and test datasets used similar schemas.Source: §2 · p. 2
  • Because phishing URLs were roughly 1 in 50 thousand, benign URLs were down-sampled to a 1:20 phishing-to-benign ratio in training and validation.The test set contained 8,742 phishing URLs and 1,775,413 benign URLs among 1,784,155 records.Source: §2 · p. 2
  • Phishing labels came from analyst-confirmed URLs, including suspicious user reports and manually verified known-phishing lists.Benign URLs were validated by analysts or production-grade offline automated analysis, rather than being simply undetected URLs.Source: §2 · p. 2

3 METHODOLOGY

URLTran adapts transformers to phishing URL classification through two approaches: fine-tuning pretrained models and pre-training a domain-specific transformer. Its inference pipeline tokenizes a URL, generates an embedding, and classifies the resulting representation.

3 METHODOLOGY

  • URLTran explores fine-tuned BERT and RoBERTa models alongside a domain-specific transformer that is pre-trained and then fine-tuned.The domain-specific model uses vocabularies built with different tokenization approaches.Source: §3 · p. 3
  • At inference, a subword tokenizer extracts URL tokens, a transformer generates an embedding, and a classifier predicts a phishing score.The predicted score indicates whether the URL corresponds to a phishing web page.Source: §3 · p. 3
  • The methodology also evaluates the best model under adversarial settings and trains it with adversarial examples to improve robustness.The paper describes these settings after the transformer training tasks.Source: §3 · p. 3

3.1 Architecture

URLTran converts URLs into subword tokens, processes them with a transformer, and uses the resulting representation for classification. Its tokenization choices balance compact full-word representations with subword information, while a dense layer supports classification and masked-token prediction.

3.1 Architecture

  • The architecture converts a URL into a numerical representation for downstream processing, with transformer details treated as background to the section.The section describes tokenization schemes and the overall classification architecture.Source: §3.1.1 · p. 3, §3.1 · p. 3

3.1.1 Tokenization.

  • Subword tokenization can reveal shared URL morphology, such as the common “bank” in “bankofamerica” and “bankofcanada,” while retaining character subsequences.The paper uses WordPiece for URLTran_BERT, BPE for URLTran_RoBERTa, and custom character- and byte-level BPE vocabularies for URLTran_CustVoc.Source: §3.1.1 · p. 3

3.1.2 Classifier.

  • The classifier applies a dense two-class layer to the pooled [CLS] representation, while masked-language pre-training uses a vocab_size-class layer to predict masked tokens.The final dense layer’s output is a phishing score for classification or masked-token probability score vectors for language modeling.Source: §3.1.2 · pp. 3–4

3.2 Training

Training combines URL masked-language-model pretraining with URL-classification fine-tuning. Fine-tuning uses the model-specific pretrained or URL-pretrained weights as initialization.

3.2 Training

  • 15% of tokens were selected for masking during URL masked-language-model pretraining, with dynamic masking and the training subset used to prevent data leakage.Of selected tokens, 80% were replaced with [MASK], 10% left unchanged, and 10% replaced by a random vocabulary token.Source: §3.2.1 · p. 4

3.2.2 Fine Tuning.

  • URLTran_BERT and URLTran_RoBERTa fine-tuning started from their authors’ large internal natural-language-corpus parameters, while URLTran_CustVoc used its MLM-learned weights.Source: §3.2.2 · p. 4
  • URL classification fine-tuning improved model parameters using gradient descent with Adam and cross-entropy loss.Source: §3.2.2 · p. 4

3.3 Adversarial Attacks and Data Augmentation

The section constructs adversarial phishing URLs through homoglyph substitutions and English compound-word splits, and uses query-parameter reordering as benign augmentation. Augmented datasets retain original URLs and labels where appropriate.

3.3 Adversarial Attacks and Data Augmentation

  • The simulated attacks modify benign URLs using patterns based on frequently observed phishing attacks, while the generated domains are absent from the pre-existing training and testing data.Source: §3.3 · p. 4

3.3.1 Homoglyph Attack.

  • Homoglyph attacks replace one randomly selected domain character with a visually similar Latin or Cyrillic Unicode character, and the resulting URL is labeled phishing.Source: §3.3.1 · p. 4

3.3.2 Compound Attack.

  • Compound attacks split domains into English dictionary words joined by hyphens, with dynamic programming finding valid splits case-insensitively.Source: §3.3.2 · p. 4

3.3.3 Parameter Reordering.

  • Parameter reordering permutes query parameters to create benign examples because the reordered URL remains valid and does not represent a phishing attack.Source: §3.3.3 · p. 4
  • For each URL, augmentation was applied with probability 0.5; selected URLs received one of the three transformations with equal probability, and the original was also retained.Source: §3.3.4 · p. 5

4 NUMERICAL EVALUATION

The evaluation compares URLTran variants with prior URL detectors at low false-positive rates, examines custom-vocabulary hyperparameters and adversarial robustness, and reports training and inference times.

Setup

  • The evaluation used TPR at low FPR thresholds because the large class imbalance makes accuracy a poor performance metric.Previously published baselines were configured according to their original papers, and URLTran_CustVoc settings varied layers, sequence lengths, tokenizer types, and vocabulary sizes.Source: §4 · p. 5

URLTran_CustVoc

  • Across 15 sampled URLTran_CustVoc settings, the three-layer model generalized best; smaller byte-level vocabularies tended to perform better overall, while sequence length had little effect.At low FPR, vocabulary-size differences were not significant; the authors hypothesize that limited pre-training data and epochs favored fewer layers.Source: §4 · p. 5, Figure 3 · p. 5

Model Performance

  • At 0.01% FPR, URLTran_BERT reached 86.80% TPR versus 71.20% for URLNet and 52.15% for Texception; URLTran variants matched or exceeded URLNet across 0.001%-0.75% FPR.The ROC plots focus on the low-FPR region, and Table 2 compares performance metrics.Source: §4 · p. 6, Figure 4 · p. 6, Figure 5 · p. 6

Adversarial Evaluation

  • With adversarial attack patterns added to training, URLTran adapted to novel attacks and outperformed its unprotected version, whereas URLTran_BERT’s performance dropped under attack.Figure 6 compares the unprotected model under attack with adversarially augmented training.Source: §4 · p. 6, Figure 6 · p. 6

Training and Inference Times

  • The best URLTran_BERT model took 4:57:11 to train on an NVIDIA V100 and averaged 0.36096 milliseconds per sample at inference.The reported total inference time was 0:10::44.Source: §4 · p. 6

5 RELATED WORK

The related work situates URLTran among transformer-based language models, text-attack methods, and URL-focused phishing and malicious-page detectors.

Text Embeddings

  • Transformer-based embeddings grew out of NLP work, including BERT [31], GPT, GPT-2, GPT-3 [5], and RoBERTa [23].RoBERTa used optimization and training-method changes to improve on BERT.Source: §5 · p. 7

Adversarial Attacks on Text

  • Text adversarial methods include white-box Hotflip and black-box frameworks such as DeepWordBug and TextAttack; URLTran specializes a threat-model-based black-box scheme for URLs.The cited approaches aim to preserve semantic constraints without changing the underlying text label.Source: §5 · p. 7

URL-Based Phishing and Malicious Web Page Detection

  • Other URL-detection approaches include lexical-feature online learning, page-content CNNs, LSTMs, and capsule networks.The paper distinguishes its URL-only processing from page-content extraction and notes the computation and memory costs of LSTMs on long URLs.Source: §5 · p. 7
  • URLNet [21] combines character- and word-level CNNs, while Texception uses separate CNNs with multi-size text windows and FastText or Word2Vec embeddings.Texception can combine blocks and adaptive max-pooling layers in configurations of different depths and widths.Source: §5 · p. 7

6 CONCLUSION

URLTran applies transformers to classify URLs as phishing or benign, and the paper reports improved detection at very low false positive rates. It also reports robustness to novel attacks under specific threat models when training uses adversarially augmented data.

6 CONCLUSION

  • URLTran predicts whether an unknown URL references a phishing or benign web page.Source: §6 · p. 7
  • URLTran uses BPE tokenizers and transformer embeddings, rather than extracted lexical features or CNN kernels spanning multiple characters and words, with a dense linear layer as input classifier.Source: §6 · pp. 7–8
  • URLTran significantly outperforms recent baselines across a wide range of very low false positive rates and can be made robust to novel attacks under specific threat models.The robustness result uses adversarially augmented training data.Source: §6 · p. 8

A HYPERPARAMETER SETTINGS

The appendix provides hyperparameter settings for three URLTran variants and two baseline models to support reproducibility.

A HYPERPARAMETER SETTINGS

Loading 2106.05256v3…