Source-linked AI summary
Nebulon Enterprise Simulated Threats for Phishing Research (NEST-Phish): A Synthetic Enterprise Phishing Email Dataset for Behavioral and Machine-Learning Research
Emily J. Winokur, Lauren S. Treiman, Allen G. Moore, Paul Schutte, Joe Ingram, Danielle N. Sanchez
TL;DR
Publicly shareable datasets combining realistic enterprise email themes, matched phishing and legitimate messages, controlled cue variation, and human and machine evaluation remain limited. The paper introduces a synthetic Nebulon dataset and evaluates it with human judgments and classifiers, finding meaningful variation in phishing judgments and learnable structure for supervised detection.
Problem
Publicly shareable datasets combining enterprise-like themes, matched phishing and legitimate emails, controlled cue variation, and human and machine evaluation remain limited.
Source: §2 · p. 3Method
The authors built a synthetic enterprise email corpus with matched scenarios and controlled phishing cues, then evaluated it through human judgments and classifier benchmarks.
Source: §1 · p. 2, §7 · p. 21Results
Human judgments show meaningful variation, while classifier benchmarks show learnable structure for supervised phishing detection.
Source: §7 · pp. 20–21Takeaways & Limitations
The corpus provides a controlled, publicly shareable resource for studying phishing detection, human susceptibility, and benchmark performance in enterprise-like contexts.
Source: §9 · p. 23The emails are fully synthetic, and the online individual-email task lacked many contextual cues present in real inboxes and workflows.
Source: §7.1 · p. 22
Abstract
from arXiv · showhide
Phishing remains one of the most persistent cyber threats, yet publicly shareable datasets for studying phishing in realistic enterprise email settings remain limited. To address this gap, we introduce a synthetic enterprise phishing email dataset built around a fictitious organization, Nebulon. The dataset spans a broad set of workplace communication themes and includes matched synthetic legitimate and phishing emails with interpretable phishing-cue annotations. Here, ``legitimate'' denotes the non-phishing class, not legitimately occurring organizational emails. Human-subject categorizations and classifier evaluations show that the dataset supports meaningful variation in phishing judgments while also providing learnable signal for supervised detection. This publicly released resource is intended to support future work on phishing detection, human susceptibility, explainability, and benchmark development in enterprise-like contexts.
1 Introduction
Phishing remains a persistent threat, while human judgment still matters despite technical defenses. The paper addresses the shortage of shareable enterprise-like datasets for both human and machine evaluation with a synthetic Nebulon corpus.
1 Introduction
- In 2024, phishing was the FBI Internet Crime Complaint Center’s most frequently reported crime, with 193,407 complaints.The passage describes phishing as a persistent threat and a primary vector for credential theft, fraud, and broader cyber compromise.Source: §1 · pp. 1–2
- Technical defenses can reduce exposure but do not eliminate phishing, leaving human judgment an important line of defense.The authors motivate datasets that support studying how people interpret, misclassify, and respond to suspicious emails.Source: §1 · p. 2
- Publicly shareable datasets combining enterprise-like themes, matched phishing and legitimate emails, controlled cue variation, and human and machine evaluation remain limited.The paper introduces a synthetic dataset built around the fictitious organization Nebulon to address this gap.Source: §1 · p. 2
2 Related Work and Existing Datasets
Prior public datasets support phishing, spam, and email-classification research, but the reviewed resources do not combine the specific features needed for matched enterprise phishing judgments by people and machines.
2 Related Work and Existing Datasets
- The reviewed public datasets span website and URL benchmarks, email and spam corpora, phishing and fraud collections, merged datasets, and synthetic resources.The review notes that its coverage is representative rather than exhaustive.Source: §2 · p. 2, Table 1 · p. 3
- To the authors’ knowledge, no publicly shareable dataset combines enterprise-like themes, matched phishing and legitimate messages, controlled cue variation, and validation in both human and machine settings.The paper positions this combination as the need motivating its work.Source: §2 · pp. 2–3
- Existing resources serve valuable but different tasks; some lack matched phishing and legitimate emails or enterprise contexts, while synthetic resources may lack realistic themes, controlled cue variation, or human evaluation.Website- and URL-focused datasets do not capture email-level linguistic and contextual cues; some email collections contain only one class.Source: §2 · pp. 2–3
3 Present Work and Contributions
The paper presents a publicly shareable synthetic enterprise email dataset designed for behavioral research and machine-learning benchmarking. It combines matched scenarios and controlled cue variation with human and classifier evaluations.
3 Present Work and Contributions
- The dataset uses SME input and LLM-assisted generation from a prior synthetic corpus, with iterative review to improve realism, reduce repetition, and balance cue prevalence.Legitimate and phishing emails were generated in matched pairs sharing a scenario, tone, and subject matter while differing in phishing-related structure.Source: §3 · p. 4
- Online human judgments and supervised classifier evaluations assess whether the emails capture variation in classification difficulty and contain learnable structure.Participants classified emails and rated confidence; classifier evaluation tests the dataset’s benchmarking use.Source: §3 · p. 5
- The corpus is intended to support phishing detection, human susceptibility, explainability, and benchmark development, not replace naturally occurring organizational email.Its contribution is a controlled, shareable resource for enterprise-like contexts.Source: §3 · p. 5
4 Dataset Construction & Validation
The dataset was built from a structured seed, enterprise themes, and an interpretable cue taxonomy, then generated and reviewed iteratively. Manual downselection yielded a 146-email corpus for participant and classifier analyses.
4.1 Seed Data and Design Goals
- The seed dataset supplied structured email content and cue annotations, while design goals called for realistic workplace messages, controlled difficulty, roughly balanced matched classes, and synthetic release.“Legitimate” denotes the non-phishing class, not naturally occurring organizational email.Source: §4.1 · pp. 5–6
4.2 Themes
- The theme set was iteratively expanded to cover varied workplace contexts, from finance, IT, HR, compliance, and operations to travel, wellness, training, and physical access.The themes were intended to represent routine benign communications and plausible phishing attempts embedded in business workflows.Source: §4.2 · pp. 6–7
4.3 Phishing Signal Taxonomy
- The taxonomy combines six interpretable binary indicators with realistic textual and structural cues, which were not all encoded as separate dataset columns.The indicators are not deterministic phishing markers: links, attachments, generic greetings, and urgent language can also occur in legitimate messages.Source: §4.3 · pp. 7–8
4.4 LLM-Assisted Email Generation
- ChatGPT 5.2 generated one legitimate-phishing pair at a time from structured examples, with human review and revision between pairs.Pairs shared scenario and contextual framing, while prompts and editing varied difficulty and allowed legitimate messages to contain superficially suspicious features.Source: §4.4 · pp. 8–9
4.5 Internal Review and Quality Control
- Internal reviewers categorized emails, rated confidence, and flagged ambiguity, implausibility, repetition, weak pair differentiation, or cue-label inconsistencies for revision or removal.Iterative review improved realism, internal consistency, and difficulty variability before participant assessment.Source: §4.5 · pp. 9–10
4.6 Initial Downselection of Candidate Emails
- After manual review of 169 candidate emails to remove artifact-driven ambiguity while preserving challenging but coherent examples, 23 were excluded, leaving 146 for all subsequent participant and classifier analyses.Excluded items included overly ambiguous or internally contradictory cues, sender–link combinations plausibly legitimate or conflicting with the intended label, misspellings or domain-formatting artifacts, and unclear organizational context.Source: §4.6 · p. 10
4.7 Final Dataset Characteristics
- 146 retained emails comprise 77 legitimate and 69 phishing messages across 17 enterprise themes, with binary cue annotations and cue overlap between classes.The six indicators are links, attachments, generic greetings, urgent language, external senders, and requests for personal information.Source: §4.7 · p. 11, Table 4 · p. 11, Table 5 · p. 12
5 Human Evaluation
The human evaluation used trained participants to classify enterprise-themed emails and rate their confidence. Accuracy varied by true email label and cue count, while theme-level differences were descriptive and confidence tracked correctness.
5.1 Participant Recruitment and Characteristics
- Among 30 U.S.-based participants, the mean age was 35.1 years; all had completed at least 10 Prolific submissions and had approval ratings above 90%.Participants were paid $12 and spent a mean of 65.5 minutes; procedures received human-subjects-board approval and participants gave informed consent.Source: §5.1 · p. 12
5.2 Participant Training
- Participants trained on nine phishing indicators and completed a seven-question quiz before classifying emails.Grammatical errors were included in training but were not strongly represented in the dataset, which aimed for plausible enterprise-style phishing.Source: §5.2 · pp. 12–13
5.3 Procedure
- Each participant classified 172 randomly ordered emails and rated confidence from 0–100; the retained set yielded 4,380 trial-level judgments.The 172 included 85 legitimate, 84 phishing, and three attention-check emails; all participants passed the checks, and accuracy feedback was not provided.Source: §5.3 · p. 13, Figure 1 · p. 14
5.4 Overall Participant Performance on the Refined Set
- 82.1% overall accuracy masked a label gap: participants correctly classified 88.8% of legitimate emails and 74.7% of phishing emails.They mislabeled 25.3% of phishing emails as legitimate, versus 11.2% of legitimate emails as phishing.Source: §5.4 · p. 14, Figure 2 · p. 15, Table 6 · p. 15
5.5 Performance Varies with Phishing-Related Signal Count
- Each additional signal was associated with lower odds of correct classification for legitimate emails (OR = 0.68, p = .004), but higher odds for phishing emails (OR = 2.14, p < .001).The interaction between signal count and true email label was significant (χ2(1) = 20.09, p < .001).Source: §5.5 · p. 16, Figure 3 · p. 16
5.6 Performance Across the Enterprise Themes
- Theme had no significant overall effect on accuracy (χ2(16) = 14.60, p = .554), although legitimate emails were more accurately classified in 16 of 17 themes.Survey / prize promotion was the exception; the largest label differences occurred in physical access/security, data privacy/records management, and training/certifications.Source: §5.6 · p. 17, Table 7 · p. 17
5.7 Confidence and Classification Accuracy
- Correct responses received 9.18 points more confidence than incorrect responses, while confidence did not differ significantly by true email label.The correctness-confidence relationship was similar for phishing and legitimate emails; the interaction was not significant (χ2(1) = 1.89, p = .169).Source: §5.7 · p. 18, Figure 4 · p. 18
6 Classifier
The study evaluated logistic regression and random forest classifiers using structural, text, and combined email features. Combined features generally performed best, and random forest outperformed logistic regression in the evaluated configurations.
6 Classifier
- The six model configurations paired Logistic Regression or Random Forest with structural-only, text-only, or combined features.Text features used TF-IDF representations of sender, subject, and body, while structural features were binary indicators.Source: §6 · pp. 18–19
- Combined structural and text features generally produced the strongest performance; Random Forest outperformed Logistic Regression overall, but the comparison is proof-of-concept because only two classifiers were tested.For Random Forest, the combined-feature improvement over text-only features was small.Source: Table 8 · p. 19, §6 · p. 19
7 Discussion
The discussion presents Nebulon as a controlled synthetic enterprise-email resource for human and machine phishing research, while emphasizing that synthetic data and limited evaluations bound what the results establish.
7 Discussion
- The dataset spans workplace themes including finance, compliance, IT, travel, document sharing, physical access, and corporate announcements.Broader thematic coverage may improve realism because phishing plausibility can depend on organizational routines and workflows.Source: §7 · p. 20
- Participants performed above chance, but phishing emails were harder to classify; additional cues helped identify phishing and increased legitimate-email misclassification.The cue effects suggest that interpretation depended on message and scenario context rather than simple rules.Source: §7 · pp. 20–21
- Matched phishing and legitimate messages vary phishing cues while overlapping in scenario, reducing topic-level confounds without guaranteeing equal empirical difficulty.The resulting examples range from obvious attacks to ambiguous legitimate messages with superficially suspicious features.Source: §7 · p. 20
- Both classifiers performed well above chance, combined text and structural features generally performed best, and Random Forest outperformed Logistic Regression overall.Only two model families were evaluated, so these benchmarks demonstrate learnability rather than an optimal modeling approach.Source: §7 · p. 21
- Generation began from a prior synthetic dataset and used expert input, an explicit cue taxonomy, matched-pair prompting, and iterative review to improve plausibility and reduce repetition.Manual downselection removed examples whose ambiguity arose from construction artifacts, balancing realism with experimental control.Source: §7 · pp. 21–22
7.1 Limitations
- The emails are fully synthetic, and stylistic regularities, differences from real enterprise email, or latent LLM tendencies may remain despite editing and review.The generation process was intended to reduce these issues but cannot eliminate them entirely.Source: §7.1 · p. 22
- Human judgments came from individual online evaluations, while the modest sample, sparse theme and cue coverage, and compact taxonomy limit the evaluation scope.Classifier results are proof-of-concept: only two model families and limited feature representations were tested, and generalization to naturally occurring enterprise emails was not demonstrated.Source: §7.1 · p. 22
8 Public Release
The public release provides a retained 146-email dataset and accompanying files for reuse in phishing benchmarking and behavioral analyses.
8 Public Release
- The release and reported analyses use the final retained set of 146 emails, although participants evaluated a broader 169-email candidate set.The dataset is released through Kaggle.Source: §8 · p. 22
- Five files provide the seed dataset, final email dataset, email-level human summaries, anonymized participant responses, and a data dictionary.The final email dataset contains the 146 retained emails used for analysis and release.Source: §8 · pp. 22–23
- The files support detection benchmarking, behavioral and item-level difficulty or confidence analyses, and transparency about the structured generation starting point.All emails concern fictitious Nebulon rather than a real company or operational environment, and participant identifiers are anonymized.Source: §8 · p. 23
9 Conclusion
The conclusion presents a controlled, publicly shareable synthetic enterprise phishing dataset for behavioral research and machine-learning benchmarking.
9 Conclusion
- The dataset combines realistic workplace themes, matched legitimate and phishing scenarios, controlled cue variation, and validation with human subjects and classifiers.It is designed for behavioral phishing research and machine-learning benchmarking.Source: §9 · p. 23
- The dataset is a useful controlled resource for studying phishing detection, human susceptibility, and benchmark performance in enterprise-like contexts.Its scope is enterprise-like contexts rather than naturally occurring organizational email.Source: §9 · p. 23
- The authors state that the dataset is not a substitute for naturally occurring organizational email.Source: §9 · p. 23
10 Funding
The article was authored by Sandia employees under a U.S. Department of Energy contract, with rights and public-access terms stated.
10 Funding
- The article was authored by employees of National Technology & Engineering Solutions of Sandia, LLC under Contract No. DE-NA0003525 with the DOE.Source: §10 · p. 23
- The employee owns the article’s rights and is solely responsible for its contents.Source: §10 · p. 23
- The U.S. Government retains a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the article for government purposes.The DOE will provide public access to the results under its Public Access Plan.Source: §10 · p. 23
A Training Modules
The appendix describes an interactive phishing-awareness module and its comprehension quiz, then reports email signal-count distributions and participant accuracy by theme and true label. Participants classified legitimate emails more accurately than phishing emails in most themes, with the difference varying across workplace contexts.
A Training Modules
- The training interface presented nine interactive tips; participants had to open every tile before proceeding, and clicking a tile displayed its information.Figure A.1 depicts the module interface and its interaction sequence.Source: Figure A.1 · p. 26, A Training Modules · p. 26
A.1 Training Comprehension Quiz
- The comprehension quiz covers phishing cues including generic or absent greetings, spelling errors, lookalike sender addresses, urgent language, requests for personal details, double-extension attachments, and HTTP links.Its stated correct answers identify these as phishing indicators or explain the security difference between HTTP and HTTPS.Source: §A.1 · pp. 27–28
A.2 Signal Count Distribution Across Legitimate and Phishing Emails
- Table A.1 reports retained-email distributions by phishing-related signal count and true email label, with percentages calculated within each label.The caption does not provide the distribution values.Source: Table A.1 · p. 29
A.3 Accuracy by Theme and True Label
- Legitimate emails were classified more accurately than phishing emails in most themes, though the size of the difference varied across workplace contexts.Figure A.2 shows mean participant accuracy by enterprise theme and true email label, with 95% confidence intervals across participants.Source: Figure A.2 · p. 30