Malla: Demystifying Real-world Large Language Model Integrated Malicious Services

1 Introduction

• The study collected 212 real-world Malla samples and used measurement and reverse-engineering tools to examine the ecosystem and its modus operandi.Its questions cover ecosystem participants, orchestration and monetization, and techniques for exploiting LLMs.Source: §1 · p. 1
• Mallas grew rapidly from April 2023 to October 2023 over six months; certain services generated compilable malicious code, while EscapeGPT produced operational phishing-site code that went unnoticed.Malla malicious-code generation cost $5–199 versus $399 from traditional malware vendors; WormGPT revenue exceeded $28K in two months.Source: §1 · p. 2
• Miscreants used uncensored LLMs and jailbreak prompts against public APIs; the study identified 182 prompts targeting five APIs, with OpenAI most frequently targeted.gpt-3.5-turbo appeared particularly susceptible to jailbreak prompts.Source: §1 · p. 2
• The paper presents the first in-depth empirical study of real-world cybercriminal activity involving LLMs offered as malicious services.It characterizes development frameworks, exploitation techniques such as jailbreak prompts, and the quality of generated malicious content.Source: §1 · p. 2
• The released artifacts include 45 prompts used to engineer malicious code and phishing campaigns and 182 jailbreak prompts that circumvent public LLM protections.The study also reports abuse of public LLMA hosting platforms for hosting Mallas.Source: §1 · p. 2

2.1 Large Language Model

• In “pre-train and prompt,” users guide an LLM with prompts, the predominant Malla approach; “pre-train and fine-tune” adapts a model using task-specific labeled data.Prompting is described as straightforward and cost-effective, while fine-tuning can be computationally expensive.Source: §2.1 · p. 2
• Vendors provide LLM APIs and safety measures, while hosting platforms let users deploy custom models and prompts; the study finds safeguards can be circumvented and platforms exploited.Usage policies prohibit activities such as malicious code generation, fraud, and scams.Source: §2.1 · p. 3

2.2 Threat Model

• The threat model assumes a miscreant with ordinary-user capabilities and no special privileges, who offers malicious services by bypassing safety measures or wrapping uncensored LLMs.Examples include generating malicious code, phishing emails, and phishing or scam webpages.Source: §2.2 · p. 3
• A Malla provider deploys a public API or uncensored model, hosts and promotes the service, then sells access to users who generate malicious content through a GUI or API.Deployment may use a web service or third-party LLMA hosting platform.Source: §2.2 · p. 3, Figure 1 · p. 3
• The study defines malicious service as LLM misuse for cybercrime and focuses on malicious code, phishing emails, and phishing or scam webpages.It acknowledges that LLMs may be misused for other prohibited purposes but centers its study on cybercrime.Source: §2.2 · p. 3

3.1 Malla Services

• The service search covered 13,353 listings from nine underground marketplaces and forums, using 145 LLM-related keywords and manual verification of GPT-4-filtered candidate listings.Collection spanned November 30, 2022 to October 12, 2023 and focused on malicious-code and cyber-product/service marketplaces rather than illegal-drug markets.Source: §3.1 · p. 4
• The commercial-service collection comprised 25 listings and 14 identified services, with nine service samples used to investigate backend LLMs.The study also extracted 45 malicious prompts from listing screenshots; backend identification used source-code inspection, traffic analysis, or hosting-page parsing.Source: §3.1 · pp. 4–5

3.2 Malla Projects

• Projects were found by searching Poe and FlowGPT with 73 keywords, then testing suspicious projects with malicious prompts and retaining those producing at least one valid response in three queries.A valid response contains properly formatted, compilable or readable malicious content; invalid examples include uncompilable code or unusable phishing pages.Source: §3.2 · pp. 5–6
• 198 public Malla projects were collected—125 from Poe and 73 from FlowGPT—alongside 127 visible jailbreak prompts and five identified backend LLMs.The project dataset also included 184 projects capable of malicious code generation, 80 of phishing emails, and 31 of phishing web pages.Source: §3.2 · p. 6

3.3 Discussion

• The authors caution that reliance on observed services, fingerprintable models and prompts, and accessible resources may bias findings and warrants caution in drawing conclusions.The study received institutional review board approval and used guidelines including requesting voucher copies and not deanonymizing sellers.Source: §3.3 · p. 6

4.1 Scope and Magnitude

• 93.40% of Mallas offered malicious code generation, 41.51% phishing email crafting, and 17.45% scam website creation among 14 services and 198 projects.Each service and project averaged more than one malicious functionality.Source: §4.1 · p. 6
• 24 of 25 service listings and 69.60% of Poe and 86.30% of FlowGPT projects remained available during the study period.Only four of nine services were operational or accessible by March 2024; service unavailability was attributed to model deprecation or hosting-site shutdowns.Source: §4.1 · p. 7

4.2 Malla Stakeholders

• The study identified 11 Malla vendors, 109 Poe project owners, and 54 FlowGPT project owners; some vendors also used dedicated servers, Poe demos, and storefronts hosted on public web platforms.FlowGPT project usage averaged 10,603.32, compared with 3,845.93 for a manual sample of 100 popular non-Malla projects.Source: §4.2 · pp. 7–8, §4.1 · p. 7

4.3 Price Strategy and Revenue

• WormGPT generated $28,325 from 27 Bitcoin and 57 Ethereum incoming transactions between July 20 and September 12, 2023, averaging approximately $15,965 per month.Malla services used fixed and subscription pricing, with prices ranging from $5 to $199 per month; jailbreak-prompt-based services were often cheaper than fine-tuned-based services.Source: §4.3 · p. 8

5.1 Methods and Metrics

• The quality metrics assess format, compilability, validity, readability, and evasion, but do not measure actual effectiveness or maliciousness.The study separately used exploit case studies and human-subject research to assess those outcomes.Source: §5.1 · p. 9

5.2 Results and Findings

• EscapeGPT and DarkGPT led malicious-code generation, with compilability rates of 66.67% and 65.08%; neither service’s compile-ready code was detected by VirusTotal, except one DarkGPT sample.The flagged DarkGPT sample was a ransomware response; Evil-GPT and WolfGPT followed closely in code generation.Source: §5.2 · p. 9
• WolfGPT led phishing-email creation, while EscapeGPT performed best on phishing sites; only 34.48% of sites from other services passed syntax checks and ran in browsers.Every WolfGPT email passed format and readability checks; the passage attributes its higher evasion to shorter emails with fewer detected spam words.Source: §5.2 · p. 9
• Malla projects on FlowGPT and Poe showed weak phishing-email and site performance, with 48.03% and 55.25% of site requests refused, respectively.Nearly all syntactically correct, browser-executable sites evaded VirusTotal detection.Source: §5.2 · p. 10
• Manual review found malicious functionality in most sampled compilable code and valid phishing sites that evaded VirusTotal detection.Four of 30 code samples and three of 30 website samples were non-malicious.Source: §5.2 · p. 10

5.3 Case Study: Effectiveness of Malla-generated Exploit Payloads

• None of 22 compilable exploits for two specific CVEs succeeded on VICIdial, but seven exploits compromised WebGoat targets despite not matching its vulnerable code.The WebGoat exploits were described as basic and not tailored to the specific CVEs.Source: §5.3 · pp. 10–11

5.4 User Study: User Susceptibility to Malla-created Phishing

• Participants identified 78.05% of Malla-created phishing emails and 72.93% of Malla-created phishing sites as phishing, versus lower rates for non-Malla phishing.The study excluded sender addresses and URLs, which may limit how well its results generalize to real-world user behavior.Source: §5.4 · pp. 11–12

6.1 Methodology

• Backend models for four services and 143 projects were identified from source code or hosting pages; the remaining three services and 55 projects were reverse-engineered.The authors trained an authorship-attribution classifier on 15,114 responses to 20 malicious-code prompts across six candidate LLMs; five-fold cross-validation yielded 0.87 precision and recall, and identified Davinci-003, GPT-3.5, and Luna AI Llama2 Uncensored for DarkGPT, EscapeGPT, and FreedomGPT, respectively.Source: §6 · p. 12, §6.1 · pp. 12–13
• 52 jailbreak prompts associated with 54 Malla projects were recovered; the prompt-leaking attack achieved 93.01% success on visible prompts, with similarity scores of 0.88 and 0.83.The evaluation used Jaro-Winkler and semantic textual similarity, respectively.Source: §6.1 · p. 13

6.2 Abused Uncensored LLMs

• In the study’s performance assessment using its stated method and metrics, Davinci-002 and Davinci-003 outperformed Pygmalion-13B and Luna AI Llama2 Uncensored for malicious-code generation.Source: §6.2 · p. 13

6.3 Prompt Engineering on Public LLM APIs

• EscapeGPT’s recovered jailbreak framed the model as a “blackhat evil confidant,” resembling a publicly known prompt that casts an “evil confidant” as having escaped rules, policies, and ethics.The similar public prompt also parallels the vendor name “EscapeMatrix.”Source: §6.3 · p. 14
• Malicious content from public jailbreak prompts could be highly similar to content from Malla services, indicating a risk from public jailbreak prompts.Source: §6.3 · p. 14

Mitigation

• The authors recommend profiling released malicious and jailbreak prompts to improve guardrails, restricting access to uncensored models, updating alignment training, and continuously monitoring emerging Malla tactics.They identify large-scale detection of cybercriminal LLM misuse as a prerequisite to mitigation.Source: §7 · p. 14
• LLMA hosts such as Poe and FlowGPT, and storefronts such as Sellix.io and BTCPay Server, are identified as stakeholders for disruption; the authors report long-lived storefronts and gaps in monitoring.They specifically describe FlowGPT as offering unrestricted access to uncensored LLMs without clear usage guidelines.Source: §7 · pp. 14–15

Other types of fraud and abuse using Malla services

• The study focuses on malicious code and phishing emails or sites because listings emphasize these functions and provide demo evidence for them, but not for other advertised abilities.Listings also claim capabilities such as lead and misinformation generation, bank-card collection, underground-market navigation, and vulnerability detection.Source: §7 · p. 15

Real-world Malla-generated instances

• Generated malicious code had not been reported to VirusTotal, and no collected real-world phishing email or webpage met the study’s attribution criteria.The highest cosine similarities were 0.64 for emails and 0.84 for webpages, both below the 0.9 threshold; the authors note that prompt variety, prompt volume, and available public datasets constrain this experiment.Source: §7 · p. 15

Ongoing emergence of new Mallas

• New Mallas were still appearing as of the June 2024 camera-ready submission, and their investigation was left for future work.Source: §7 · p. 15

• Earlier studies explored LLM use for misinformation, deepfake profiles, spear phishing, attack payloads, malware, and hateful memes, but the authors report that these works did not study tangible cybercriminal activity through malicious services.Source: §8 · p. 15
• Related vulnerability research covers prompt manipulation and jailbreaking; unlike prompt manipulation, jailbreaking needs no access to the model’s system prompt.The authors position their work as uncovering mechanisms underpinning Malla and supplementing understanding of LLM exploitation.Source: §8 · p. 16

• Across 14 Malla services and 198 projects, the study identified 45 malicious prompts, eight backend LLMs, and 182 jailbreak prompts, alongside ecosystem findings on development, hosting, pricing, and revenue.The authors state that these findings and artifacts will help better defend against LLM misuse for cybercriminal activities.Source: §9 · p. 16

A Performance Comparison between Uncensored Models

• Pygmalion-13B generates compilable malicious code (F = 0.66,C = 0.47), all of which evades VirusTotal detection (E = 0.47), but fails to produce valid phishing sites (F = 0.2,V = 0,E = 0).Luna AI Llama2 Uncensored has lower code capability (F = 1,C = 0.17), with compilable code evading VirusTotal (E = 0.17); its phishing sites are also invalid (F = 1,V = 0,E = 0).Source: A Performance Comparison between… · pp. 20–21
• Davinci-002 and Davinci-003 slightly outperform Pygmalion-13B and Luna AI Llama2 Uncensored on three malicious-code metrics and generate more valid, stealthy phishing sites.Davinci-002 also surpasses the others in evading phishing-email detector scans.Source: A Performance Comparison between… · p. 21
• All four models produce readable phishing emails at rates exceeding 85%, but Pygmalion-13B and Luna AI Llama2 Uncensored fail to evade OOPSpam (E = 0 and E = 0.13, respectively).Pygmalion-13B emails have an average spam-word count of 7.04, substantially above WolfGPT's metrics.Source: A Performance Comparison between… · pp. 20–21

Comparison with Public Jailbreak Prompts

• The comparison tests 96 operational public jailbreak prompts combined with malicious prompts against gpt-3.5-turbo, yielding 3,908 code-containing responses, and compares them with 146 Malla responses.Each combined prompt was queried three times; Jaccard similarity measured code-segment similarity.Source: B Comparison with Public Jailbreak… · p. 21
• Malla-generated malicious code consistently has higher Jaccard similarity to code from public jailbreak prompts than to code from other Malla services.MakerGPT and XXXGPT score 0.8421 and 0.8393, respectively, against jailbreak-prompt code.Source: B Comparison with Public Jailbreak… · pp. 21–22
• XXXGPT's generated code matches the jailbreak-prompt code exactly except for comments.The passage also reports high similarity scores of 0.8421 for MakerGPT and 0.8393 for XXXGPT.Source: B Comparison with Public Jailbreak… · p. 22