PaperlayerView exact revision on arXiv

Source-linked AI summary

A Large-Scale Empirical Study of Modern Phishing Email Content

Jaehwan Park, Woonghee Lee, Fujiao Ji, Doowon Kim

arXiv:2609.30683v1cs.CR

TL;DR

Prior phishing-content characterizations relied on bag-of-words models that capture lexical co-occurrence but miss the semantics that make lures persuasive. This study measures phishing emails across message text and attachments, examining theme, CTA, impersonation, and their associations. It finds diverse themes but concentrated CTAs, with CTA and impersonation patterns linked to theme.

  • Problem

    Prior characterizations used bag-of-words models that capture lexical co-occurrence but not the semantics that make phishing lures persuasive.

    Source: §I · p. 1

  • Method

    The study analyzes 2.9M real-world phishing emails across theme, CTA, and impersonation, pairing message text with images, PDFs, and calendar invitations.

    Source: §I · p. 2

  • Results

    73.0% of emails use URL navigation, while no theme exceeds 21.3%; CTA and impersonation are theme-dependent, and invoice phishing shifted toward offline communication.

    Source: §I · p. 2, §IX · p. 13

  • Takeaways & Limitations

    Images supplement message text, PDFs carry the pretext, and calendar invitations reinforce interaction endpoints, leaving text-only detection with little to inspect in a substantial fraction of emails.

    Source: §I · p. 2

    Prevalence estimates characterize the APWG-reported corpus, not all phishing emails in the wild, and may reflect APWG reporting practices and vantage points.

    Source: B. Limitations · p. 13

Abstract

from arXiv · show

Phishing remains one of the most pervasive threats to Internet users, and email remains its predominant delivery channel. Email content is the attack surface of phishing: it is what the victim reads and what automated defenses inspect. Yet the composition of modern phishing content is poorly measured. Prior work has characterized dimensions such as theme, call-to-action (CTA), and impersonation, but not at scale, and their associations and temporal changes remain unclear, owing to small or source-specific corpora, bag-of-words topic models, and a focus on text alone. We present a content-focused measurement study of 2.9M distinct real-world phishing emails collected over 13 months (June 2025 - June 2026) in collaboration with the Anti-Phishing Working Group (APWG). We treat each email as a composite artifact comprising message text and its attachments: 272K images, 143K PDFs, and 57K calendar invitations. Using an LLM pipeline validated against human-annotated samples, we analyze these components along three dimensions (theme, CTA, and impersonation), examine the associations among them, and measure longer-term change against a historical dataset. We find that attackers diversify what they use to deceive but converge on how victims should respond: no theme exceeds 21.3% of emails, while a single CTA, URL navigation, accounts for 73.0%. CTA and impersonation choices are conditioned on theme. Attachments play three roles: images supplement the message text, PDFs substitute for it by carrying the pretext, and calendar invitations reinforce it by replicating interaction endpoints into a persistent medium. Over the longer term, the dominant CTA for invoice-themed phishing shifted from URL navigation to offline communication, rising from 6.7% in 2015 to 46.9% in 2025.

I. INTRODUCTION

Prior work measured phishing content at limited scale and often focused on text, while the study analyzes 2.9M emails and their attachments across content dimensions and time. It finds diverse themes but theme-conditioned response strategies, distinct attachment roles, and divergent historical trajectories.

Introduction

  • Prior work relied on small or source-specific datasets, bag-of-words methods, and text-only analysis, leaving dimension associations and temporal change unquantified at scale.LDA-based characterizations capture lexical co-occurrence but not the semantics distinguishing different lures.Source: §I · p. 1
  • The study analyzes 2.9M emails collected over 13 months, pairing message text with attachments and using an LLM annotator validated against human samples.It examines theme, call to action, and impersonated entity, their co-occurrence, and changes against a historical corpus.Source: §I · pp. 1–2

Key Findings

  • No theme exceeds 21.3% of emails, while URL navigation accounts for 73.0% of CTAs, showing diverse lures but convergence on victim response.Impersonation is prevalent but dispersed across a long tail of brands.Source: §I · p. 2
  • CTA and impersonation choices are theme-conditioned: invoice emails more often prompt phone calls, while extortion commonly demands cryptocurrency.Impersonation is more common in themes borrowing organizational legitimacy, such as government and invoice, than in extortion and romance.Source: §I · p. 2
  • Images supplement message text, PDFs substitute by carrying the pretext, and calendar invitations reinforce endpoints; 97.8% of phone-bearing ICS repeat a body number.PDFs are concentrated in an Invoice/offline-communication/private-company-impersonation configuration (64.6%), while calendar reminders make repeated endpoints persistent.Source: §I · p. 2
  • Offline Communication rose from 6.7% of Invoice CTAs in 2015 to 46.9% in 2025, while Account retained a stable CTA structure as its vocabulary shifted.The study compares persistent themes over time using the Nazario corpus.Source: §I · p. 2

II. BACKGROUND

The paper treats a phishing email as message text plus attachments that may carry attack content, and studies those components alongside changes over time.

II. BACKGROUND

  • Phishing content can appear in visible message text, attachments, or both, so text-only analysis misses content carried by images, PDFs, and calendar invitations.The study compares these components with their surrounding message text.Source: §II · p. 3

III. ANALYSIS PLAN FOR PHISHING EMAIL ECOSYSTEM

  • The analysis asks how theme, CTA, and impersonation co-occur, what attachments contain, and how persistent themes change over time.The temporal comparison focuses on patterns within the same themes.Source: §III · p. 3

A. Real-world Phishing Email Collection

  • The APWG partnership collected 3.8 million emails over 13 months; deduplication and completeness checks yielded 2.9M valid emails (76.4%).The study defines message text as the subject plus visible body text, and attachments as preserved images, PDFs, and calendar invitations.Source: A. Real-world Phishing Email Collection · p. 3
  • The study targets what recipients see, using HTML when available and removing text hidden from view.Invisible text is analyzed separately in the appendix.Source: A. Real-world Phishing Email Collection · pp. 3–4
  • Extraction yielded 371,662 emails with attachments: 188,821 with images, 138,992 with PDFs, and 56,831 with calendar invitations.Reporter and recipient identifiers and other unnecessary reporting metadata were removed before submitting content for LLM analysis.Source: A. Real-world Phishing Email Collection · p. 4

B. Phishing Information Retrieval using LLM

The paper uses an LLM to classify phishing content at scale, validates it against researcher labels, and compares patterns across content types and corpora.

B. Phishing Information Retrieval using LLM

  • LLM classification addresses bag-of-words limits: word-frequency models can conflate sentences with opposing action direction, which matters for identifying phishing requests.The authors use contextual inference to extract information from emails at scale.Source: B. Phishing Information Retrieval… · p. 4
  • For images, theme and CTA classification is limited to content-bearing documents and banners, while impersonation also includes logos.The pipeline first assigns images to functional roles; prompt instructions also tell the model to classify, not follow, embedded commands.Source: B. Phishing Information Retrieval… · p. 4
  • GPT-5.1 achieves the highest message-text F1 across theme (0.988), CTA (0.990), and impersonation (0.984), so the authors use it for the remaining analysis.Two researchers independently labeled a random sample of 1,000 emails and retained labels agreed upon by both.Source: B. Phishing Information Retrieval… · p. 5
  • GPT-5.1’s attachment validation F1 scores are 0.970 for image functional-role classification, 0.951/0.975/0.975 for image content, and 0.979/0.989/0.995 for PDFs.The image-content scores correspond to theme, CTA, and impersonation; PDF scores use the same three labels.Source: Table III · p. 4, B. Phishing Information Retrieval… · p. 5

C. Analysis Methodology

  • Historical comparisons focus on Account and Invoice, the themes with sufficient volume across corpora, and adjust for corpus differences using overlapping 2025 samples and size-matched subsamples.Attachment comparisons with the Nazario corpus are directional rather than quantitative because few attachments were recovered.Source: C. Analysis Methodology · pp. 5–6

IV. RQ1: CATEGORIZING PHISHING EMAIL CONTENT

Across phishing themes, requested actions are far more concentrated than narrative pretexts; the section also identifies emails without an identifiable theme and non-URL response paths.

1) Themes:

  • No theme exceeds 21.3% of emails; Promotion leads at 21.3%, followed by Invoice at 16.2% and Account at 12.9%.Five leading themes together account for 72.3%, while ten other themes make up a long tail.Source: 1) Themes · p. 6
  • 5.3% of emails have insufficient content to identify a theme, despite being confirmed phishing through URLs flagged by at least four VirusTotal engines.Extortion combines relatively short subjects with among the longest bodies.Source: 1) Themes · p. 6

2) CTAs:

  • 73.0% of emails request URL navigation, while 22.3% request a non-URL action or pair one with a URL.Offline communication accounts for 13.6%; URL suppression leaves the secondary action intact in emails that pair a URL with another action.Source: 2) CTAs · pp. 6–7
  • URL-only analysis misses emails whose sole requested action is non-URL, and does not fully stop emails combining URL navigation with another action.The authors identify phone numbers, wallet addresses, attachments, and reply-to addresses as additional detection signals; observed cryptocurrency payments are a conservative estimate because Monero transactions are not publicly observable.Source: 2) CTAs · pp. 6–7

3) Impersonation:

Impersonation is prevalent but differs in concentration by entity type, while both CTA and impersonation patterns vary with phishing theme.

3) Impersonation:

  • Company impersonation appears in 63.5% of emails and government impersonation in 4.1%; 32.3% contain no message-content impersonation.These rates cover recipient-facing email content, not impersonation introduced only after a recipient follows an embedded URL.Source: 3) Impersonation · p. 7
  • Government impersonation is concentrated among fewer entities, while company impersonation spans tens of thousands of brands.The top 100 entities cover 57.3% of government-impersonation emails and 43.8% of company-impersonation emails, unlike phishing webpages, where the top 100 brands exceed 90%.Source: 3) Impersonation · p. 7, Figure 3 · p. 7

1) Theme–CTA Associations:

  • URL navigation ranges from 94% of Romance emails to 18% of internal-business emails, despite accounting for 73.0% overall.Invoice favors offline communication (59%), extortion cryptocurrency payment (51%), and internal business email replies (60%).Source: 1) Theme–CTA Associations · pp. 7–8, Figure 4 · p. 8

2) Theme–Impersonation Associations:

Impersonation rates and entities vary by phishing theme. Images usually reinforce message-text pretexts, but sometimes supply attack content omitted from the text.

2) Theme–Impersonation Associations:

  • Government, Survey, and Invoice themes show high impersonation, while Extortion and Romance show substantially lower rates.The leading impersonated entities align with theme pretexts, including the Social Security Administration in Government, Costco in Survey, and PayPal in Invoice.Source: 2) Theme–Impersonation Associations · p. 8, Figure 5 · p. 8

2) Images in the Message Text:

1) Characteristics of Images:

  • Image delivery channels carry different content: data-URI images tend toward branding, attachments toward self-contained visual artifacts, and CID images mix both.The corpus contained 272,124 images across 188,821 emails; CID references were the most common channel (66.6%).Source: 1) Characteristics of Images · pp. 8–9, Table VI · p. 9

2) Images in the Message Text:

  • Image roles often match message-text themes, such as human photographs in Romance and document images in Shipping and Invoice.Romance images were 81% Profile/Human photos; Document/Letter images made up 58% of Shipping and 56% of Invoice images.Source: 2) Images in the Message Text · p. 9, Figure 6 · p. 9

B. PDFs: Primary Carriers of Phishing Content

PDFs often carry phishing content absent from message text and cluster around an Invoice–Offline Communication–Company configuration. Calendar invitations usually repeat interaction endpoints and can resurface them as reminders.

B. PDFs: Primary Carriers of Phishing Content

  • PDFs supply themes, CTAs, and impersonation absent from message text in 28.0%, 79.0%, and 85.4% of PDF-bearing emails, respectively.When the message text lacks one of these elements, an attached PDF almost always supplies it; PDF-bearing emails also have substantially shorter bodies.Source: B. PDFs: Primary Carriers of Phishing… · p. 10, Table VII · p. 10
  • Invoice, Offline Communication, and company impersonation account for 73.2%, 83.8%, and 90.8% of PDF content, respectively.Their combined configuration covers 64.6% (92,495) of PDFs.Source: B. PDFs: Primary Carriers of Phishing… · p. 10, Table VII · p. 10
  • PDF phishing concentrates on a narrow attack configuration but uses a dispersed pool of callback numbers.Among company-impersonating PDFs, PayPal accounts for 52.2%; the ten most frequent phone numbers cover only 10.7% of phone-bearing PDFs.Source: B. PDFs: Primary Carriers of Phishing… · p. 10

C. ICS: Reinforcing Interaction Endpoints

  • ICS attachments contain phone numbers in 81.4% of cases and URLs in 25.5%, usually repeating endpoints already present in message text.They can also supply a missing endpoint and resurface a phone number through a calendar display reminder.Source: C. ICS: Reinforcing Interaction… · p. 11

VI. RQ3: TEMPORAL COMPARISON

A comparison bridged the contemporary corpus to the historical Nazario corpus for the persistent Account and Invoice themes. Their trajectories differ: Account remains comparatively stable in CTA and impersonation, while Invoice undergoes a major CTA shift alongside distinct lexical changes.

A. Historical Corpus Characterization

  • Historical non-text coverage is sparse: Nazario contains only 94 PDFs and 19 Document/Letter images, unevenly distributed across years.These components support cross-corpus comparison but not standalone longitudinal inference.Source: A. Historical Corpus Characterization · p. 11

B. Bridging Historical and Modern Corpora

  • The overlapping 2025 samples show similar within-theme structure across the contemporary and Nazario corpora for CTA, impersonation, and lexical characteristics.CTA divergences were smaller across corpora than between Account and Invoice within either corpus.Source: B. Bridging Historical and Modern… · p. 11

C. Temporal Changes within Persistent Themes

  • Invoice’s dominant CTA shifted to Offline Communication in 2025, reaching 46.9% from 6.7% in 2015, while Account’s URL Navigation remained dominant at 90.2–98.7% annually.Invoice impersonation remained comparatively stable; Account showed no significant consecutive-year CTA changes after correction.Source: C. Temporal Changes within Persistent… · p. 12

D. Lexical Change Within Persistent Themes

  • Account vocabulary shifted from mailbox administration toward cloud-stored content despite stable CTA and impersonation patterns.Historical terms include “mail” and “server,” while contemporary terms include “cloud,” “photos,” and “storage.”Source: D. Lexical Change Within Persistent… · p. 12
  • Invoice shows more limited lexical change alongside its CTA shift, with modern messages more often including transaction details such as amount and date.Historical Invoice messages more often used “download.”Source: D. Lexical Change Within Persistent… · p. 12

VII. RELATED WORK

The paper contrasts its large-scale, cross-component analysis with prior work limited by corpus, method, and unit of analysis. It also draws defensive implications while bounding its prevalence and historical claims to the observed data and available themes.

VII. RELATED WORK

  • Prior studies offered limited contemporary visibility, focused mainly on text, and did not systematically quantify content associations at scale.Several also used bag-of-words methods that provide limited semantic context.Source: §VII · pp. 12–13

A. Mitigation

  • Defenses should inspect attachments and non-URL interaction paths, and consider theme-aware configurations rather than isolated indicators.PDFs can supply CTAs absent from text, while calendar invitations can preserve and re-surface interaction endpoints.Source: A. Mitigation · p. 13
  • Invoice phishing shifted toward offline communication, while Account phishing retained a comparatively stable CTA structure, supporting periodic reevaluation of detection models and training scenarios.Within Invoice, offline communication rose from 6.7% to 46.9% of CTAs and became dominant in 2025.Source: A. Mitigation · p. 13

B. Limitations

  • Prevalence estimates describe APWG-reported emails rather than all phishing, and the historical comparison covers only Account and Invoice.The longitudinal analysis may also reflect differences between the APWG and Nazario corpora; historical attachment coverage is sparse.Source: B. Limitations · p. 13

IX. CONCLUSION

  • Across 2.9M emails, themes are diverse while CTA and impersonation choices are associated with theme; attachments supplement, substitute for, or reinforce message text.Invoice phishing shifted toward offline communication, whereas Account phishing retained a comparatively stable CTA structure.Source: §IX · pp. 13–14

ETHICS CONSIDERATIONS

  • The study removes reporting metadata and unnecessary recipient-identifying information before LLM analysis, reports aggregate results, and sanitizes examples.It does not release the raw corpus or live phishing URLs, and the corpus remains subject to APWG’s data-sharing agreement.Source: ETHICS CONSIDERATIONS · p. 14

APPENDIX B QUERY AND PROMPTS

The appendix describes query prompts for classifying phishing content and the rules used to constrain those judgments. It also identifies macro-averaged validation as treating each class equally.

APPENDIX B QUERY AND PROMPTS

  • The prompts classify theme, requested action (CTA), impersonation, and image functional role using category choices and definitions.The prompt template is described as incorporating the categories and definitions in Table XIII.Source: Appendix B · p. 16
  • Classification rules require explicit evidence, prohibit guessing or relying on excluded context, and select the primary lure except when multiple action types are explicitly requested.For impersonation, the content must explicitly present itself as an organization; merely mentioning one is insufficient.Source: Appendix B · p. 16
  • The prompt specifies a JSON-only output format and an impersonation rule requiring explicit organizational self-presentation.The output is structured as a JSON array.Source: Appendix B · p. 16
  • Macro-averaged validation gives each class equal weight in reported precision, recall, and F1.The table caption describes the metric weighting, not the scores themselves.Source: Table IX · p. 17

APPENDIX C MACRO-AVERAGED VALIDATION PERFORMANCE

Macro-averaged validation reports strong GPT-5.1 performance across theme, CTA, and impersonation. The appendix also describes variation in message lengths and cryptocurrency payment and address patterns.

APPENDIX C MACRO-AVERAGED VALIDATION PERFORMANCE

  • 0.967, 0.971, and 0.963 are GPT-5.1 macro-F1 scores for theme, CTA, and impersonation, respectively.Each class contributes equally, and the results indicate performance is not driven solely by prevalent classes.Source: Appendix C · p. 17

APPENDIX D SUBJECT AND BODY LENGTHS ACROSS THEMES

  • 2,272 characters is Extortion’s longest median body length, while Shipping has the shortest median body length.Extortion emails can carry a threat, cryptocurrency instructions, and a deadline; Shipping emails use compact transaction details.Source: Appendix D · p. 17, Figure 9 · p. 18

APPENDIX E ANALYSIS OF CRYPTOCURRENCY ADDRESSES

  • 438 of 2,539 unique wallet addresses (17.2%) appear in at least two emails and account for 75.3% of address occurrences.The most frequent address appears in 465 distinct emails.Source: Appendix E · p. 17
  • Bitcoin has 1,612 unique addresses across 6,531 occurrences, while Litecoin emails carry nearly one unique bech32 address per email.The Litecoin count is 917 bech32 addresses over 1,070 occurrences; Table XII includes 919 addresses overall.Source: Appendix E · p. 17

APPENDIX D SUBJECT AND BODY LENGTHS ACROSS THEMES

  • 20 and 28 characters are the median subject lengths for Internal Business and Extortion, respectively.Subject lengths are similar across most themes, with these two as notable differences.Source: Appendix D · p. 17, Figure 9 · p. 18

APPENDIX E ANALYSIS OF CRYPTOCURRENCY ADDRESSES

  • ρ = 0.02 (n = 170) indicates negligible rank correlation between Bitcoin address reuse and revenue; funded Bitcoin addresses averaged 24.0 occurrences versus 1.71 for unfunded ones.In a four-chain example, only the Bitcoin address was funded; the Litecoin, Ethereum, and Tron addresses remained empty.Source: Appendix E · pp. 18–19
  • 215 of 2,539 addresses (8.5%) received payments, totaling $414,979 across 816 transactions, with Bitcoin accounting for 87.7%.The median funded address collected $1,396, and the ten largest addresses together held 25.1% of proceeds.Source: Appendix E · p. 18, Table XII · p. 19

APPENDIX F PHISHING IMAGE EXAMPLES

The appendix gives representative images for several functional-role categories. The examples include images assigned no discernible role, logo-only images, and document or promotional images.

APPENDIX F PHISHING IMAGE EXAMPLES

  • Figure 10 shows an image categorized as “No Discernible Role.”The appendix presents it as a representative example of that category.Source: Appendix F · p. 19
  • In this section, Figures 11 and 12 show two forms of “Logo Only”: an emblem-style logo and a wordmark, respectively, as representative examples of each image role.Source: Appendix F · p. 19
  • Figures 13 and 14 show “Document/Letter” and “Advertisement/Promotional Banner” images, respectively.Source: Appendix F · p. 19

APPENDIX G ANALYSIS OF THEME, CTA, AND IMPERSONATION BY

Document/Letter and Ad/Promo Banner images exhibit distinct theme–CTA configurations, while historical phishing distributions show persistent Account prominence and shifting themes and impersonation patterns. GPT-5.1 is used to classify content after evaluation against human-annotated samples.

IMAGE ROLE

  • Document/Letter images concentrate on Invoice (72.7%) and Offline Communication (76.3%), while Ad/Promo Banner images center on Promotion (78.1%) and URL navigation (69.0%).Both roles predominantly use private-company impersonation (86.5% and 72.9%, respectively); “No explicit action” is more common in banners than documents (23.7% versus 12.3%).Source: IMAGE ROLE · p. 19, Table XIV · p. 22, Table XV · p. 22
  • Across 2015–2025, Account remained the largest theme each year (42–79%), while Invoice generally accounted for 10–25%.Themes and impersonation patterns shift over time, while URL navigation remains the dominant call to action.Source: IMAGE ROLE · p. 19, Figure 15 · p. 20

APPENDIX I USE OF GENERATIVE AI

  • GPT-5.1 classifies message text, images, and PDFs by theme, call-to-action, and impersonation, with image functional role also classified, after evaluation against human-annotated samples.The authors retain responsibility for the taxonomy, prompts, methodology, analysis, interpretation, and claims.Source: Appendix I · p. 19
Loading 2609.30683v1…