Source-linked AI summary
To Ensemble or Not: Assessing Majority Voting Strategies for Phishing Detection with Large Language Models
Fouad Trad, Ali Chehab
TL;DR
LLM outputs vary with prompt design and model training, while research on combining multiple LLMs for classification remains limited. This study evaluates prompt-based, model-based, and hybrid majority voting for phishing URL detection. Ensembling helps when components perform similarly, but may not outperform the strongest component when their performance differs substantially.
Problem
Research has largely improved individual LLMs, leaving limited evidence on combining multiple LLMs to reduce response variance and improve phishing URL classification.
Source: §1 · pp. 1–2Method
Results
Ensembling improves performance when participating components have comparable performance but may not surpass the highest-performing component when performance differs substantially.
Source: §6 · p. 11, §7 · p. 12Takeaways & Limitations
Ensemble selection should depend on component performance parity rather than applying ensembling uniformly.
Source: §6 · p. 11, §7 · p. 12The study identifies fixed ensembles as limited for diverse data or tasks and suggests dynamic model selection as future work.
Source: §7 · p. 12
Abstract
from arXiv · showhide
The effectiveness of Large Language Models (LLMs) significantly relies on the quality of the prompts they receive. However, even when processing identical prompts, LLMs can yield varying outcomes due to differences in their training processes. To leverage the collective intelligence of multiple LLMs and enhance their performance, this study investigates three majority voting strategies for text classification, focusing on phishing URL detection. The strategies are: (1) a prompt-based ensemble, which utilizes majority voting across the responses generated by a single LLM to various prompts; (2) a model-based ensemble, which entails aggregating responses from multiple LLMs to a single prompt; and (3) a hybrid ensemble, which combines the two methods by sending different prompts to multiple LLMs and then aggregating their responses. Our analysis shows that ensemble strategies are most suited in cases where individual components exhibit equivalent performance levels. However, when there is a significant discrepancy in individual performance, the effectiveness of the ensemble method may not exceed that of the highest-performing single LLM or prompt. In such instances, opting for ensemble techniques is not recommended.
1 Introduction
This study addresses limited research on combining LLMs by introducing and evaluating three majority-voting ensembles for phishing URL classification. It finds that ensembling helps when components perform similarly, but may not outperform the strongest component when performance differs substantially.
Motivation
- The research targets a gap in literature that has emphasized improving individual LLMs rather than combining multiple models to reduce response variance and improve accuracy.Prior work focuses on architecture, training data, prompt engineering, and fine-tuning.Source: §1 · p. 2
Contributions
- The study introduces prompt-based, model-based, and hybrid majority-voting strategies for LLM text classification.Prompt-based voting varies prompts for one LLM, model-based voting combines multiple LLMs on one prompt, and hybrid voting varies both.Source: §1 · p. 2
Experimental scope
- Experiments use zero-shot, one-shot, and two-shot prompts across GPT-3.5-Turbo, GPT-4, Gemini-1.0-pro, PaLM 2, and LLaMA 2 (70B).These prompt types and five LLMs are evaluated with the proposed ensemble techniques.Source: §1 · p. 2
2 Background and Preliminaries
The background situates the study in transformer-based LLMs, evolving phishing threats, and ensemble prediction. It motivates majority voting as a way to combine model decisions for phishing URL detection.
2.1 Large Language Models (LLMs)
- LLMs are neural-network models whose transformer architecture and scale support language tasks including translation, sentiment analysis, and text generation.The transformer architecture introduced by Vaswani et al. is presented as foundational to modern LLM development.Source: §2.1 · p. 3
- ChatGPT, LLaMA, PaLM, and Gemini exemplify transformer-based LLMs increasingly used as off-the-shelf systems for diverse applications.These models extend the trend toward deploying pretrained language systems without extensive end-user training or maintenance.Source: §2.1 · p. 3
2.2 Phishing URL Detection
- Phishing URLs use deceptive domains, trusted brands, homoglyphs, valid TLS certificates, logos, shortening, and redirection to evade users and detection.These evolving tactics create a need for advanced URL analysis methods.Source: §2.2 · p. 3
2.3 Ensemble Strategies
- Majority voting determines a classification from the majority decision of multiple models and is intended to improve accuracy by aggregating independent predictions.The study applies this ensemble approach to LLM-based phishing URL classification.Source: §2.3 · p. 3
3 Related Work
Prior phishing detection progressed from blacklists and feature-based machine learning to deep learning and prompt-engineered LLMs. However, LLM studies largely used single-model, single-prompt settings, leaving ensemble combinations underexplored.
Phishing URL Detection
- Prompt-engineered LLMs achieved promising phishing-detection results but did not surpass fine-tuned models in the cited studies.Those studies were conducted within single-model, single-prompt frameworks.Source: §3 · p. 4
- Traditional and machine-learning phishing detectors analyze URL structure, lexical features, tokens, special characters, length, and subdomains using methods such as SVM, decision trees, and Random Forest.Character-level deep neural networks later extended these approaches.Source: §3 · p. 4
Ensemble Methods
- Although majority voting, bagging, and boosting aggregate predictions effectively in simpler models, ensemble use remains less investigated in NLP and modern LLM text classification.The literature therefore provides limited evidence on combining LLM predictions.Source: §3 · p. 4
Research Gap
- This study addresses the gap by introducing and evaluating three majority-voting ensemble strategies for phishing URL text classification.The strategies are designed to assess how multiple LLMs’ collective strengths can improve reliability and accuracy.Source: §3 · p. 4
4 Methodology
The methodology defines three majority-voting designs that vary prompts, models, or both for classifying the same URL. The hybrid design combines model diversity with prompt variation.
Method Overview
- The methodology evaluates three ensemble strategies designed to improve phishing URL detection with LLMs.The strategies are presented as prompt-based, model-based, and hybrid approaches.Source: §4 · p. 5
Prompt-based Ensemble
- The prompt-based ensemble gives one LLM multiple prompt variations for the same URL and majority-votes its resulting classifications.Figure 1a illustrates one LLM receiving three different prompts.Source: §4 · p. 5
Model-based Ensemble
- The model-based ensemble gives the same prompt to multiple LLMs and majority-votes their independent decisions.Figure 1b depicts three LLMs analyzing one prompt.Source: §4 · p. 5
Hybrid Ensemble
- The hybrid ensemble sends multiple prompts to multiple LLMs and aggregates all responses through majority voting.It is intended to combine the benefits of model diversity and prompt variation.Source: §4 · p. 5
5 Experiments
The experiments evaluate majority-voting ensembles for phishing URL detection using a balanced PhishStorm subset, five LLMs, and zero-, one-, and two-shot prompts. Accuracy and F1-score are compared with individual-component baselines.
5.1 Experimental Setup
- 1,000 PhishStorm URLs—500 phishing and 500 legitimate—form the balanced evaluation subset.Source: Dataset · p. 5
LLMs:
- Five chat-based LLMs are evaluated: GPT-3.5-Turbo, GPT-4, Gemini 1.0 Pro, LLaMA 2 (70B), and PaLM 2.Source: LLMs · p. 5
- Each LLM classifies batches of 50 URLs using a single prompt per batch.Source: Prompts · p. 5
Prompts:
- The prompt-based ensemble aggregates one LLM’s responses across multiple prompts by majority voting.Source: Prompts · p. 6, Figure 1 · p. 6
- Zero-shot prompts provide no examples, one-shot prompts provide one example, and two-shot prompts provide phishing and legitimate examples.Source: Prompts · p. 6
- Accuracy and F1-score are compared against individual ensemble components to assess majority-voting benefits.Source: Evaluation Framework · p. 6
Evaluation Framework:
The evaluation establishes model–prompt baselines for phishing URL classification before assessing ensembles. GPT-4 is strongest, while other models show differing sensitivity to prompt design.
5.2 Individual LLMs Performance
- GPT-4 achieves the highest accuracy, 0.946 with one-shot prompting, and an F1-score of 0.943.Source: §5.2 · p. 7
- LLaMA 2 records the lowest reported peak performance, with 0.830 accuracy and 0.797 F1-score under two-shot prompting.Source: §5.2 · p. 7
- Gemini and PaLM 2 remain consistent across prompt types, whereas GPT-3.5-Turbo varies substantially with prompt design.Source: §5.2 · p. 7
- GPT-3.5-Turbo improves from 0.854 zero-shot accuracy to 0.879 one-shot accuracy, then declines to 0.856 with two-shot prompting.Source: §5.2 · p. 7
5.3 Prompt-based Ensemble
The ensemble experiments compare majority voting with the best individual prompt or model. They show that aggregation is most useful when the combined components perform similarly.
5.3 Prompt-based Ensemble
- Prompt-based ensembling matches or improves the best single-prompt performance for every model except GPT-3.5-Turbo.Source: §5.3 · p. 8, Figure 3 · p. 8
- GPT-3.5-Turbo declines under prompt-based voting because two lower-performing prompts outweigh its stronger one-shot prompt.Source: §5.3 · p. 8
- Prompt-based ensembling is most effective when prompts produce similar performance for a given LLM.Source: §5.3 · p. 8
5.4 Model-based Ensemble
- Model-based ensembling gives every model the same prompt and aggregates their URL predictions by majority vote.Source: §5.4 · p. 8
- Across five models, model-based voting does not surpass GPT-4 because its superior performance is diluted by lower-performing models.Source: §5.4 · p. 8, Figure 4 · p. 9
- The similarly performing model ensemble improves over individual models across all prompt types.Source: §5.4 · p. 9
- When component performance differs substantially, selecting the highest-performing model produces better results than model-based voting.Source: §5.4 · p. 9
5.5 Hybrid Ensemble
The hybrid ensemble combines multiple prompts and multiple LLMs through majority voting. Including all five models fails to improve on GPT-4, whereas a more balanced Gemini–PaLM ensemble improves performance.
5.5 Hybrid Ensemble
- The hybrid method aggregates predictions from multiple LLMs receiving various prompts.Source: §5.5 · p. 9
- Including all five models does not improve over GPT-4, the highest-performing individual model.Source: §5.5 · p. 9, Figure 7 · p. 11
- GPT-4’s superior performance across prompts skews the full hybrid ensemble toward lower-performing models.Source: §5.5 · p. 10
- Excluding GPT-4, LLaMA 2, and variable GPT-3.5-Turbo yields a balanced ensemble containing Gemini and PaLM.Source: §5.5 · p. 10
- The refined Gemini–PaLM hybrid shows significant improvement when models and prompts have similar performance levels.Source: §5.5 · p. 10
6 Discussion
Ensembling is most effective when participating models perform similarly; large performance gaps can let one model dominate and eliminate the benefit. The paper therefore recommends evaluating individual and ensemble performance before deployment.
6 Discussion
- Ensembling is most effective when participating models perform at similar levels, whereas widely varying performance can provide no real benefit.A higher-performing model may disproportionately affect the ensemble, while comparable models can yield notable improvement.Source: §6 · p. 11
- Before production deployment, practitioners should assess individual models and ensemble configurations because the available LLM pool is limited and performance varies.This evaluation is intended to identify configurations that improve rather than diminish overall system performance.Source: §6 · p. 11
- Figure 8 examines hybrid-ensemble performance among models and prompts with similar performance levels.Source: Figure 8 · p. 12
7 Conclusion
The study concludes that ensemble methods can improve phishing URL detection when component models have relatively similar performance, but may not help when one model is substantially stronger or weaker. It proposes adaptive, richer voting, and larger-scale studies as future directions.
7 Conclusion
- Ensembling improves phishing URL detection when models have relatively similar performance but may not help when one model is significantly better or worse.Source: §7 · p. 12
- Dynamic ensembling could adaptively select models according to the task or input data, addressing limitations of fixed ensembles across diverse settings.Source: §7 · p. 12
- Future work could develop voting mechanisms beyond simple majority rule, including weighted voting based on model confidence or past performance.Source: §7 · p. 12
- Larger-scale studies should include more LLMs, including models trained on specialized datasets, to examine ensembling across broader language-understanding capabilities.Source: §7 · pp. 12–13