PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Dark Side of AI: Advanced Phishing Attack Design and Deployment Using ChatGPT

Nils Begou, Jeremy Vinoy, Andrzej Duda, Maciej Korczynski

arXiv:2309.10463v1cs.CRcs.AI

TL;DR

The paper examines the potential for ChatGPT to be misused to develop and deploy phishing attacks. It uses ChatGPT to generate phishing infrastructure and evaluates the resulting sites. The assessment reports swift generation and deployment and close resemblance to target websites.

  • Problem

    Potential malicious exploitation of ChatGPT is a concern, including its use in social engineering and sophisticated scams.

    Source: §I · p. 1

  • Method

    The authors use ChatGPT to generate phishing-site code and automate infrastructure deployment, then compare the generated sites’ source code and appearance with the originals.

    Source: §I · p. 1

  • Results

    The assessment reports that ChatGPT can swiftly generate and deploy phishing websites whose pages significantly resemble the originals.

    Source: §I · p. 1, §VI · p. 6

  • Takeaways & Limitations

    The authors conclude that ChatGPT remains susceptible to malicious use despite safeguards, with potential risks to the reach and magnitude of phishing attacks.

    Source: §VI · p. 6

    ChatGPT’s response to website-code modification depends on website characteristics, and alternative approaches were sometimes explored.

    Source: C. Adapting the Website Code · p. 3

Abstract

from arXiv · show

This paper explores the possibility of using ChatGPT to develop advanced phishing attacks and automate their large-scale deployment. We make ChatGPT generate the following parts of a phishing attack: i) cloning a targeted website, ii) integrating code for stealing credentials, iii) obfuscating code, iv) automating website deployment on a hosting provider, v) registering a phishing domain name, and vi) integrating the website with a reverse proxy. The initial assessment of the automatically generated phishing kits highlights their rapid generation and deployment process as well as the close resemblance of the resulting pages to the target website. More broadly, we demonstrate that recent advances in AI underscore the potential risks of its misuse in phishing attacks, which can lead to their increased prevalence and severity. This highlights the necessity for enhanced countermeasures within AI systems.

I. INTRODUCTION

The paper investigates ChatGPT’s potential misuse in building and deploying phishing attacks, motivated by concerns about malicious exploitation. It generates phishing infrastructure and assesses how closely the resulting sites resemble their targets.

I. INTRODUCTION

  • The study is motivated by concerns that ChatGPT capabilities may be maliciously exploited for social engineering, malicious content, automated attacks, and sophisticated scams.Prior reports described ChatGPT use to develop malicious tools and plausible phishing emails,.Source: §I · p. 1
  • ChatGPT is used to generate and deploy phishing infrastructure, including cloned sites, credential-capture code, obfuscation, hosting, domain registration, and reverse-proxy integration.The paper also assesses source-code and visual similarity to the original website.Source: §I · p. 1
  • The initial assessment finds that ChatGPT can generate and deploy phishing websites swiftly, with pages resembling the targeted sites.The assessment compares the original and altered pages and evaluates website source-code similarity.Source: §I · p. 1
  • The authors describe the work as the first peer-reviewed study of generating and deploying fully functional phishing websites using ChatGPT, and report bypassing its malicious-code filters.The paper also states that it successfully generates phishing-site code and automates deployment.Source: §I · p. 1

II. BACKGROUND ON PHISHING ATTACKS

The section outlines phishing techniques and toolkits, then frames the threat posed by their combination. Phishing can span site imitation, infrastructure, credential handling, evasion, and distribution.

A. Overview of Phishing Techniques and Toolkits

  • Phishing campaigns combine site imitation, infrastructure acquisition, credential storage and transmission, evasion, and fraudulent-URL distribution to support large-scale attacks –.The paper describes phishing kits as packaged resources for replicating sites and collecting information, which can enhance campaign effectiveness and scalability.Source: A. Overview of Phishing Techniques and… · p. 2
  • Phishers imitate target sites using techniques such as URL spoofing, cloning, HTML/CSS manipulation, content scraping, SSL certificates, and dynamic content to deceive users into sharing sensitive information.Source: A. Overview of Phishing Techniques and… · p. 2
  • Attackers may host fraudulent sites on compromised or paid/free services, register similar domains, and use false registrant details to conceal their identities.Source: A. Overview of Phishing Techniques and… · p. 2
  • Credential theft may involve local storage or transmission through email, file-sharing services, or encrypted messaging applications such as Telegram.Source: A. Overview of Phishing Techniques and… · p. 2
  • Evasion tactics include obfuscation, URL changes or shorteners, time-based triggers, IP rotation, and human-verification mechanisms –,.Phishing URLs are commonly distributed through deceptive spam emails.Source: A. Overview of Phishing Techniques and… · p. 2

B. Threat Model

The threat model assumes an attacker with fundamental Python skills and access to specified AI models who automates a phishing kit and its infrastructure. The workflow includes site cloning, hosting, domain registration, TLS, a reverse proxy, and credential transmission.

B. Threat Model

  • The assumed attacker has fundamental Python skills and access to ChatGPT GPT-3.5-turbo-16K and OpenAI Codex models, using the OpenAI API to automate tasks.Codex is integrated with GitHub Copilot for IDE use.Source: B. Threat Model · p. 2
  • The modeled workflow clones a target site, obfuscates its content, deploys a phishing kit on a hosting platform, and registers a potentially similar domain with a TLS certificate.Source: B. Threat Model · p. 2
  • The attacker may use a reverse proxy for anonymity and a private Telegram channel to transmit stolen credentials, with Python 3.11 used for the tasks.The code is pre-compiled on a hosting server before deployment.Source: B. Threat Model · p. 2

III. ETHICAL CONSIDERATIONS

The authors describe ethical safeguards and a method for generating phishing infrastructure with ChatGPT. They divide functionality across prompts to evade filters, while acknowledging the need for safeguards against malicious use.

III. ETHICAL CONSIDERATIONS

  • The study frames understanding potential AI abuse and developing safeguards as necessary responses to risks from malicious use.Source: §III · p. 2
  • The authors withheld their specific prompts and detailed implementation information to mitigate the risk of malicious actors using the published results.Source: §III · p. 2

A. Circumventing ChatGPT Filters

  • ChatGPT filtering and content safeguards are described alongside jailbreak attempts intended to circumvent them.The paper identifies the “Do Anything Now” master prompt as one tactic for trying to bypass safeguards.Source: A. Circumventing ChatGPT Filters · p. 3
  • To evade ChatGPT detection, the authors split phishing-kit functionality across distinct prompts, each appearing harmless alone, then linked the generated components.The methodology concerns creating and deploying the phishing infrastructure using ChatGPT.Source: A. Circumventing ChatGPT Filters · p. 3, §IV · p. 3

B. Cloning a Website

The authors use ChatGPT to clone targeted websites and adapt the copies while attempting to preserve their appearance and functionality. They report a token limit for large sites and describe modifications to login forms and page code.

B. Cloning a Website

  • ChatGPT generated Python code that copied a target website into a static replica and added a web server that launched after copying.The generated code could contain errors, which the authors say ChatGPT could often correct when given a request or traceback.Source: B. Cloning a Website · p. 3
  • The GPT-3.5-turbo-16K limit of 16,384 tokens makes cloning large phishing websites challenging.Source: B. Cloning a Website · p. 3

C. Adapting the Website Code

  • For adaptation, the authors submitted copied-site source code to ChatGPT for optimization while asking it to preserve appearance and functionality.They report improved performance, faster processing for later modifications, and lower resemblance to the original source code, which could complicate code-similarity detection.Source: C. Adapting the Website Code · p. 3
  • The authors asked ChatGPT to redirect login forms to their API and add a deceptive modal; it could maintain the site's appearance and produce convincing modal text.Their approach varied with website characteristics, sometimes requiring an alternative method to extract the login form.Source: C. Adapting the Website Code · p. 3

D. Website Code Obfuscation

The authors addressed code obfuscation by prompting ChatGPT for techniques and processing site code in segments. They also report generating a credential-collection API that sent login details to Telegram.

D. Website Code Obfuscation

  • When given a whole page, ChatGPT sometimes returned truncated output, so the authors split the code into parts and submitted them individually.ChatGPT initially declined to obfuscate code directly, prompting the authors to ask about techniques and request sample code.Source: D. Website Code Obfuscation · p. 3
  • Segmenting source code into 800-character chunks avoided context-size constraints and expanded the range of obfuscation methods ChatGPT produced.Source: D. Website Code Obfuscation · p. 3
  • Obfuscation results varied between runs and could impair page functionality, including by removing login buttons or introducing inconsistent IDs.Source: D. Website Code Obfuscation · p. 4

E. Collection of Credentials

  • ChatGPT generated a Flask API from one prompt that collected login and password details through a GET endpoint and sent them to Telegram using a direct HTTP request.The implementation avoided a third-party Telegram library; the authors also set up a Telegram bot to receive credentials.Source: E. Collection of Credentials · p. 4

F. Automated Deployment of the Toolkit

The authors automated packaging and deployment of the phishing kit to cloud instances using scripts and provider APIs. They report that ChatGPT produced a functional installation script on its first attempt.

F. Automated Deployment of the Toolkit

G. Domain Name Registration

The authors used ChatGPT to support domain registration and reverse-proxy configuration. They report that generated registration details varied with prompt language, while initial identity outputs repeatedly returned “John Doe.”

G. Domain Name Registration

  • After randomly generated domain names were rejected by registrar detection, the authors used three hyphen-separated English words and a random number; they avoided typosquatting due to potential detection by anti-phishing engines.ChatGPT generated the registration class from the registrar's quick-start example.Source: G. Domain Name Registration · p. 4
  • ChatGPT initially returned “John Doe” consistently for test identity attributes, which the authors considered potentially suspicious.Source: G. Domain Name Registration · p. 4
  • Prompt language affected generated registration data: another language yielded more authentic results, and the authors randomized country choices in a Python function.Source: G. Domain Name Registration · p. 4
  • For reverse-proxy integration, ChatGPT provided a method to change nameservers and generated code that optimized supplied source code without considering potential malicious intent.Source: G. Domain Name Registration · p. 4

H. Reverse Proxy and CDN

The paper reports automated reverse-proxy configuration and phishing-site generation, alongside tests of generation rates and the time needed for page processing.

H. Reverse Proxy and CDN

  • Figure 1 compares the page layout of the original site with the kit outcome.The paper presents it in the reverse-proxy and CDN section.Source: Figure 1 · p. 5
  • The provider's reverse proxy concealed the malicious server's real IP address and supplied a TLS certificate for client connections.The authors used Flexible mode, which encrypted the client-to-CDN connection but not the CDN-to-server connection.Source: H. Reverse Proxy and CDN · p. 5
  • Given the provider's library documentation in its prompt, ChatGPT generated correct code to automate adding a domain to the reverse-proxy provider.The authors noted that the documentation was up to date, unlike the model's knowledge base.Source: H. Reverse Proxy and CDN · p. 5

A. Testing the Generation of Phishing Sites

  • 25 of 80 popular-brand sites (31.25%) were generated successfully; failures were attributed to the model's prompt token limit.The authors suggest larger token limits or code segmentation could help handle larger sites.Source: A. Testing the Generation of Phishing… · p. 5
  • The combined page-creation, minification, modal, and obfuscation steps took approximately 4 minutes on average.Basic-page creation averaged 29 seconds, minification 57 seconds, modal addition 59 seconds, and obfuscation 14 seconds per 800-character chunk.Source: A. Testing the Generation of Phishing… · p. 5

B. Case Study: the LinkedIn Site

The LinkedIn case study measures visual and source-code similarity across kit stages, while deployment tests report their success rate and completion time.

B. Case Study: the LinkedIn Site

  • LinkedIn visual similarity was 100% after copying, 99.6% after authentication changes, 90.9% after modal addition, and 87.6% after obfuscation.The authors report shifted text and missing CSS tags after obfuscation.Source: B. Case Study: the LinkedIn Site · p. 5
  • LinkedIn source-code similarity fell from 93% after adaptation to 56.6% after ChatGPT obfuscation, measured with TF-IDF.The paper describes lower code similarity as favorable for reducing detection chances.Source: B. Case Study: the LinkedIn Site · p. 6

C. Testing the Generation and Deployment

VI. CONCLUSIONS

The authors report that ChatGPT can support automated phishing-kit creation and deployment, while identifying token limits as a constraint and warning of misuse risks.

VI. CONCLUSIONS

  • ChatGPT made it possible for a person with little programming or development skill to create and deploy a fully automated phishing kit.The authors state that the person still needs to understand phishing and how to bypass ChatGPT filters.Source: §VI · p. 6
  • The case study showed high-quality generated phishing pages with significant similarity to the original website.The authors also report deploying several instances automatically within minutes.Source: §VI · p. 6
  • The prevailing token limit constrains phishing-kit generation and deployment for larger websites.The authors suggest larger-token models or code segmentation as possible mitigations, with segmentation requiring deeper expertise.Source: §VI · p. 6
  • Despite safeguards and filters, the authors conclude that ChatGPT can be used to generate and deploy phishing sites swiftly, raising potential risks and expanding phishing attacks' reach and magnitude.Source: §VI · p. 6
Loading 2309.10463v1…