PaperlayerView exact revision on arXiv

Source-linked AI summary

A Computer Vision Approach to Visual Fraud Detection in Phishing Websites Using YOLOv8

Basil Sajid Shaikh

arXiv:2609.34015v2cs.AIcs.CV

TL;DR

URL and markup signals can be rotated or obfuscated, while phishing pages rely on visual impersonation. The paper applies YOLOv8 to rendered website screenshots to detect that visual fraud. It reports 92% test accuracy and an 11% relative false-positive reduction after augmentation.

  • Problem

    URL and markup signals can be rotated or obfuscated, leaving limited evidence about whether a rendered page visually impersonates a trusted brand.

    Source: §1 · pp. 1–2

  • Method

    A YOLOv8-based detector analyzes rendered website screenshots for visual fraud indicators such as mismatched logos, login-form layouts, and brand-inconsistent structure.

    Source: §1 · p. 2

  • Results

    92% accuracy on the held-out test set, with augmentation reducing the false-positive rate by 11% relative to the pre-augmentation baseline.

    Source: §7 · p. 8, §11 · p. 11

  • Takeaways & Limitations

    The detector is positioned as a complementary signal for borderline pages or browser and email previews, not as a replacement for existing defenses.

    Source: §8 · p. 10

    Accuracy reflects a specific collection window, so periodic retraining may be needed as phishing-kit visual styles change.

    Source: §9 · p. 10

Abstract

from arXiv · show

Phishing remains one of the most common vectors for financial and identity fraud, and most detection systems still rely on inspecting a page's URL, HTML markup, or domain registration history. These signals are easy for an attacker to rotate or obfuscate, and they say very little about what actually convinces a victim to hand over a password or a card number: the way the page looks. This paper describes a visual, image-based approach to phishing detection that treats a rendered webpage the same way a human eye would, as a picture that either matches a trusted brand or doesn't. A YOLOv8 convolutional neural network was trained to classify full-page website screenshots as phishing or legitimate based on layout, logo placement, color scheme, and login-form structure, rather than on text extracted from the page. The system reached 92% classification accuracy on a held-out test set, processed a single screenshot in roughly 100 milliseconds, and, after a round of data augmentation aimed specifically at lighting, compression, and scaling variation, cut the false-positive rate by 11% relative to the pre-augmentation baseline. The paper walks through the dataset construction, the augmentation strategy, the model architecture and training setup, and the resulting performance, and closes with a discussion of where this kind of visual detector fits alongside, rather than instead of, existing URL- and content-based defenses.

1. Introduction

The paper proposes detecting phishing from rendered screenshots because URL and markup signals can be rotated or obfuscated while the impersonated brand’s visual identity remains a target. The visual detector is intended to complement existing defenses, with limited signal on pages that do not imitate a brand.

1. Introduction

  • The system assesses rendered screenshots for visual identity rather than relying on a page’s markup or address bar.It targets visual cues such as logos, login-form layouts, and brand-consistent page structure.Source: §1 · p. 2

2. Why Visual Detection

  • A brand’s visual identity changes less quickly than domains or phishing-kit markup, giving attackers an incentive to copy it faithfully.The visual cues include logo placement, colors, fonts, and login-form appearance.Source: §1 · pp. 1–2, §2 · p. 2
  • Rendered-page analysis can cover phishing pages whose credential forms are images or whose text is split across elements or embedded through JavaScript.The detector needs the page to render, not its text to be extractable.Source: §2 · p. 2
  • The visual model is framed as an additional signal for cases where URL or content checks miss, not as a replacement for those defenses.The approach is limited when a credential form copies little of a brand’s visual identity.Source: §2 · pp. 2–3

3. Related Work

Prior anti-phishing approaches use reputation checks, URL and markup features, or rendered-page imagery. This paper’s approach is closest to logo-localizing systems, but learns visual inconsistency without a maintained brand-reference database.

Related work

  • Blocklist and reputation systems are widely deployed and fast, but cannot list a domain before it has been reported.They may also use registration-age and hosting-reputation heuristics.Source: §3 · p. 3
  • URL- and markup-based systems can detect phishing without rendering pages, but do not assess what a page looks like on screen.Reported examples include a 98.2% phishing detection rate and 0.976 F1-score for Yerima and Alzaylaee [3].Source: §3 · p. 3
  • VisualPhishNet learns brand-page similarity embeddings, while Phishpedia locates logo regions and matches them against a brand database.Phishpedia reported 99.2% accuracy at roughly 0.19 seconds per page.Source: §3 · pp. 3–4
  • This system localizes screenshot regions but learns login-form, layout, and brand-color inconsistency without a separate maintained brand-matching database.Compared with reference matching, it retrains faster on new brands but gives less explicit information about which brand triggered a flag.Source: §3 · p. 4

4. System Design and Methodology

The system captures and curates full-page screenshots, preprocesses them to preserve visual proportions, and uses a fine-tuned YOLOv8 detector to produce page-level phishing scores. Training and inference choices are shaped by latency and by avoiding template leakage across data splits.

4.1 Screenshot Pipeline

  • Screenshots are captured in a headless browser at 1280 by 800 after page rendering settles, while failed, redirected, or timed-out pages are dropped.Raw screenshots are resized during preprocessing rather than at capture time.Source: §4.1 · p. 4

4.2 Dataset

  • The dataset contains a little over 20,000 labeled screenshots, with phishing captures and legitimate login pages from commonly impersonated brands.Perceptual-hash clustering limits near-duplicate phishing templates, while legitimate pages include redesigns, seasonal themes, and regional variants.Source: §4.2 · pp. 4–5

4.3 Data Preprocessing

  • Letterbox resizing preserves screenshot aspect ratio, and overlay dismissal reduces visual differences unrelated to a page’s brand identity.Images are normalized using the per-channel mean and standard deviation used for the pretrained COCO backbone.Source: §4.3 · pp. 5–6

4.4 Model

  • YOLOv8 was selected to localize logos, login forms, and other regions, whose confidence and count are combined into a page-level score.Smaller variants were chosen to meet the latency target and provide reviewers with regions behind a flag.Source: §4.4 · p. 6

4.5 Training

  • The model is fine-tuned from COCO-pretrained weights on an 80/10/10 split made by template cluster, with thresholds tuned on validation data.Reduced-precision inference takes roughly 100 milliseconds per screenshot, excluding page loading and rendering.Source: §4.5 · pp. 6–7

5. Experimental Setup

Evaluation compares raw-data and augmented models on a held-out, template-cluster-disjoint test set. It tracks classification errors and latency, with false positives emphasized because they can undermine trust in inline warnings.

Experimental Setup

  • Evaluation uses a held-out 10% test partition kept template-cluster-disjoint from training and validation.This design avoids placing near-duplicate screenshots from a template cluster across the partitions.Source: §5 · p. 7
  • The experiment compares raw-data and augmented versions under the same architecture and training schedule to test whether augmentation reduces false positives.Tracked metrics are accuracy, false-positive rate, false-negative rate, and average inference latency per screenshot.Source: §5 · p. 7

6. Augmentation Strategy

The raw-screenshot model was brittle to changes in viewport, compression, display conditions, and capture perspective. Augmentation simulated these variations, with class-balanced oversampling for less common impersonation subtypes.

6. Augmentation Strategy

  • Raw-screenshot training made the model brittle to viewport, compression, lighting, and capture variations, encouraging it to learn exact pixels rather than general layout and color.Source: §6 · p. 7
  • The pipeline simulated viewport variation with bounded random resizing and cropping, and screenshot compression with JPEG re-compression at varying quality levels.Source: §6 · p. 7
  • Brightness, contrast, saturation, rotation, and perspective changes simulated display differences and mobile or hastily taken captures.Source: §6 · pp. 7–8
  • Class-balanced oversampling targeted the less common crypto-exchange and workplace-portal impersonation subtypes, alongside the image augmentations.Source: §6 · p. 8

7. Results

On the held-out test partition, augmentation improved accuracy and reduced false positives, while inference remained around 100 milliseconds per screenshot. The remaining errors include legitimate pages during redesigns and unbranded phishing pages; comparison with published systems is approximate, with a tradeoff between accuracy and independence from a maintained logo reference list.

7. Results

  • 92% accuracy versus 87% for the baseline, and an 11% relative false-positive-rate reduction (8.1% versus 9.1%), followed augmentation on the held-out test partition.The baseline and augmented model used the same underlying images; Table 1 presents their held-out test comparison.Source: Table 1 · p. 8, §7 · p. 8
  • Most remaining false positives involved legitimate pages during redesigns or seasonal themes, whose changed logos, colors, or form layouts could resemble phishing deviations.The model's learned visual identity is only as current as its training set, so a brand redesign can temporarily look like impersonation.Source: §7 · p. 8
  • Most remaining false negatives were generic, unbranded credential forms with no visual identity for the detector to match.Source: §7 · p. 8

7.1 Comparison with Prior Visual Detection Systems

  • The published-system comparison is approximate, not a controlled head-to-head, because systems were evaluated on different datasets and protocols.Table 2 positions this work against three closely related published systems.Source: Table 2 · p. 9, §7.1 · p. 9
  • Phishpedia's reported accuracy is meaningfully higher than this work's 92%, while this approach does not depend on a maintained reference logo list.The comparison's differing datasets and protocols constrain direct interpretation; the paper says this system may need updating when its learned layout and color scheme drifts after redesign.Source: §7.1 · p. 9
  • Approximately 100 milliseconds per screenshot was faster than Phishpedia's 0.19 seconds; both were described as fast enough for inline pre-submit warnings.The paper attributes the difference largely to omitting a separate brand-database matching step; it says augmentation changes what the model learns rather than its size.Source: §7 · p. 8, §7.1 · p. 9

8. Discussion

The visual detector is intended to complement, not replace, existing anti-phishing defenses by checking pages that pass URL and content heuristics. Its speed matters for inline warnings, while brand redesigns create a need to refresh the visual reference set.

8. Discussion

  • The detector complements existing defenses by targeting pages that pass URL and content heuristics while visually reproducing a trusted brand.It may serve as a secondary check for borderline pages or as a browser-extension or email-preview signal.Source: §8 · p. 10
  • A warning must arrive before a user enters a password, making roughly 100-millisecond inference more useful for inline screening than a verdict that takes several seconds.The passage contrasts speed with a detector that is slightly more accurate but too slow for this setting.Source: §8 · p. 10
  • Brand redesigns can trigger false positives on legitimate pages until the visual reference set catches up, requiring refreshes tied to brand changes.The passage contrasts this maintenance cost with URL-blocklist systems.Source: §8 · p. 10

9. Limitations

The reported performance is bounded by the dataset's collection window and by what a single screenshot can capture. Automated capture, brand scope, and changes in phishing styles also limit what the detector can reliably see.

9. Limitations

  • The reported accuracy reflects a specific collection window, so periodic retraining is needed as phishing kit styles shift.Templates are reused, reported, and replaced over time.Source: §9 · p. 10
  • Some phishing kits detect headless-browser fingerprints and may serve scanners a different, blander page, removing the visual signal before classification.Source: §9 · p. 10
  • A single screenshot misses multi-step flows when a spoofed form appears only after a click or delay.Source: §9 · p. 10
  • Generic phishing pages without brand impersonation are close to invisible to a detector built around visual brand similarity.The passage describes this as a design boundary.Source: §9 · p. 10

10. Future Work

Future work would combine visual and URL signals, broaden brand and language coverage, and test ways to address multi-step flows and deployment latency. These changes target complementary blind spots and practical deployment constraints.

10. Future Work

  • Combining visual signals with lexical URL features in one model could bring together complementary strengths that are currently separate pipeline stages.URL features can flag newly registered suspicious domains regardless of rendered appearance, near the visual detector's blind spot.Source: §10 · p. 10
  • Periodic re-screenshotting could help detect multi-step phishing flows, but added latency must be budgeted for inline warnings.Source: §10 · p. 11
  • Expanding both datasets to include more regional brands and non-English login pages is needed before generalizing beyond the trained brand set.Source: §10 · p. 11
  • Testing a smaller, quantized model for on-device browser-extension deployment could remove network round-trip latency, making inference latency the user's end-to-end delay.The current roughly 100-millisecond inference figure is one stage of a larger pipeline.Source: §10 · p. 11

11. Conclusion

The YOLOv8 screenshot detector supports the idea that visual brand imitation can signal phishing, reaching 92% accuracy with roughly 100-millisecond inference. Its blindness to non-brand-impersonating pages supports treating it as one layer in a broader detection stack.

11. Conclusion

  • 92% accuracy, roughly 100-millisecond inference latency, and an 11% relative false-positive reduction from augmentation were reported on a deduplicated set of over 20,000 labeled screenshots.Source: §11 · p. 11
  • The detector is blind to phishing pages that do not attempt brand impersonation, supporting its use as one layer rather than a standalone solution.Source: §11 · p. 11
  • Further engineering work includes keeping brand references current, extending regional and language coverage, and reducing model size for network-free deployment.Source: §11 · p. 11

12. Conclusion

The initial research was conducted at San Diego State University under Dr. Hajar Homayouni’s supervision.

  • The initial research was carried out at San Diego State University under Dr. Hajar Homayouni’s supervision.Source: §12 · p. 11
Loading 2609.34015v2…