Source-linked AI summary
RatGPT: Turning online LLMs into Proxies for Malware Attacks
Mika Beckerich, Laura Plein, Sergio Coronado
TL;DR
Rapid plugin growth makes it difficult to track and secure plugins, while jailbreaks can circumvent some LLM safeguards. The paper demonstrates a plugin-mediated ChatGPT proxy that relays commands between a victim and an attacker’s C2 server, and reports a simulated attack that evaded the described detection systems.
Problem
Jailbreaks can circumvent some LLM safeguards, while the growing number of plugins makes it difficult to track and secure them.
Source: §1 · p. 1Method
The proof-of-concept uses ChatGPT and a web-browsing plugin as a proxy to connect a victim’s executable with an attacker-controlled C2 server.
Source: §1 · p. 1, §2.2 · p. 2Results
The authors report simulating an attack that evaded the described intrusion-detection systems because connections to an LLM are widely considered legitimate.
Source: §1 · p. 1Takeaways & Limitations
The demonstration shows that plugins can let an LLM act as a malware proxy to establish C2 connections and provide access to a victim’s machine without direct victim interaction with the LLM.
Source: §6 · p. 6Non-deterministic LLM output makes correct payload generation unreliable and cannot be guaranteed.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
The evolution of Generative AI and the capabilities of the newly released Large Language Models (LLMs) open new opportunities in software engineering. However, they also lead to new challenges in cybersecurity. Recently, researchers have shown the possibilities of using LLMs such as ChatGPT to generate malicious content that can directly be exploited or guide inexperienced hackers to weaponize tools and code. These studies covered scenarios that still require the attacker to be in the middle of the loop. In this study, we leverage openly available plugins and use an LLM as proxy between the attacker and the victim. We deliver a proof-of-concept where ChatGPT is used for the dissemination of malicious software while evading detection, alongside establishing the communication to a command and control (C2) server to receive commands to interact with a victim's system. Finally, we present the general approach as well as essential elements in order to stay undetected and make the attack a success. This proof-of-concept highlights significant cybersecurity issues with openly available plugins and LLMs, which require the development of security guidelines, controls, and mitigation strategies.
1 INTRODUCTION
LLMs have advanced software-engineering tasks but can also help generate and propagate malware, while the proliferation of plugins makes security difficult to track. The paper demonstrates a plugin-mediated ChatGPT proxy for remote control and describes how the attack can evade detection.
1 INTRODUCTION
- LLMs such as ChatGPT have achieved or exceeded state-of-the-art performance on code tasks, while their APIs also create possibilities for generating and propagating malware.The cited tasks include code summarisation, code translation, and program synthesis.Source: §1 · p. 1
- OpenAI safeguards can be circumvented by jailbreaks, and the rapid proliferation of ChatGPT plugins makes it difficult to track and secure them all.Plugins offer new ways to integrate online services into chat-based interactions.Source: §1 · p. 1
- The proof-of-concept uses ChatGPT and a plugin as a proxy between a victim and attacker-controlled web service to establish remote control and execute commands.The paper calls the resulting system RatGPT.Source: §1 · p. 1
- The simulated attack evaded the described intrusion-detection systems because connections to an LLM are widely considered legitimate.The paper contrasts this with direct victim-to-attacker connections, which those systems can detect.Source: §1 · p. 1
- The proof-of-concept keeps the payload in memory and avoids direct victim-to-attacker communication, while ChatGPT generates IP addresses and payloads on the fly.The authors report that this can bypass many malware scanners and leave no traces directly pointing to the attacker.Source: §1 · p. 1
- The authors identify an automated LLM-and-plugin command-and-control pipeline, a feasibility demonstration, and a passive attack requiring only initial execution of a seemingly harmless executable.After execution, prompts and communications proceed automatically without further victim prompts.Source: §1 · p. 1
2 APPROACH
The approach defines which plugins qualify for the attack, initializes ChatGPT to accept code-generating prompts, and uses its responses to build and run an in-memory payload. Web-browsing plugins relay commands and data between the victim and the attacker’s C2 server.
2 APPROACH
- The approach uses publicly available LLMs and plugins to generate payloads and communicate with an attacker-controlled C2 server through the LLM.Its components include prompt initialization, dynamic IP-address generation, payload generation, and C2 communication.Source: §2 · p. 2
2.1 Plugin Vulnerability Criteria
- A plugin qualifies as vulnerable for this attack if it can browse the web and visit any user-specified URL.The authors say adapting the approach to plugins that read hosted file types such as PDFs should be straightforward.Source: §2.1 · p. 2
2.2 Prompt Initialisation
- Because public LLMs have safeguards against harmful prompts, the implementation uses a modified DAN jailbreak to obtain code-compatible responses.Non-code text is surrounded with triple quotes so Python’s exec() treats it as comments.Source: §2.2 · p. 2
2.3 IP Address Generation
- 1932 was generated for the neutron-discovery year prompt, and historical-fact prompts became the method used to generate IP-address parts after mathematical prompts proved unreliable.The generated parts are prompted separately and concatenated; output is filtered to retain numbers.Source: §2.3 · p. 2
2.4 Payload Generation
- The executable combines LLM-generated IP-address and Python-code responses, then runs the code in memory to connect to the C2 server and handle its commands.The requested handlers include shell commands and file operations; in-memory code is described as harder to analyze with static methods.Source: §2.4 · p. 2, Figure 1 · p. 2
2.5 Communication with the C2 Server
- After direct GPT-4 browsing was deactivated, the approach used web-browsing plugins to relay attacker-hosted commands to the payload and return command output through URL requests.The payload polls the LLM’s web connectivity; returned website content supplies commands, and output can be appended to a URL as ASCII or Base64.Source: §2.5 · p. 3
3 PROOF OF CONCEPT
The proof-of-concept demonstrates ChatGPT acting as a proxy between a victim and an attacker’s C2 server, using an executable, ChatGPT, a VPS, and a CAPTCHA-solving service. Its scenario covers initial execution, reconnaissance, and file exfiltration through plugin-mediated requests.
3 PROOF OF CONCEPT
- The proof-of-concept demonstrates ChatGPT as a proxy between a victim and an attacker’s C2 server, while a powerful implementation is explicitly outside its scope.The authors describe the example as simple, not as evidence of functional limits.Source: §3 · p. 3
3.1 Experimental Setup
- The setup assigns ChatGPT payload generation and plugin communication, a VPS C2 hosting, the executable polling and command execution, and CAPTCHA solving to a third-party service.The executable also contains access tokens, plugin IDs, prompts, and code snippets for cleaning ChatGPT responses.Source: §3.1 · p. 3
3.2 Possible Scenario
- The scenario relies on a user executing the executable; HTTP-only outbound communication and unblocked access to online LLMs define the target network setting.The example suggests a free ChatGPT PLUS crack as a lure; closing the initial window leaves the program running in the background.Source: §3.2 · pp. 3–4
- Reconnaissance batches identity, working-directory, and directory-listing commands into one message, while later plugin polling retrieves commands and returns their output to the C2 server.The demonstration then uses shellCmd to read a deliberately placed passwords.txt file and sends its contents in a subsequent request.Source: §3.2 · p. 4
4 DISCUSSION
The discussion identifies unreliable payload generation and changing plugin availability as limitations, alongside safeguards that impede reliable use of ChatGPT as a malware proxy. It proposes future attack extensions and mitigations including website controls, LLM access restrictions, and prompt scanning.
4.1 Limitations
- Non-deterministic LLM outputs can omit required payload functions or contain parser errors, while removed or unavailable plugins can break the implementation.The authors suggest multiple web-enabled plugins as fallback options when some are removed.Source: §4.1 · p. 4
4.2 Future Work
- The authors propose prompt obfuscation because plain-text prompts expose the malware’s workings, and describe a possible ransomware example using the same process.The proposed ransomware would encrypt victim data and send the key to the C2 server through a ChatGPT-mediated request.Source: §4.2 · p. 5
4.3 Existing Safeguards
- CAPTCHA challenges are often triggered by automated payload traffic, and GPT-4’s message cap limits the number of commands an attacker could execute.Online solver services bypassed CAPTCHA protection multiple times, but bypassing suspicious-behavior detection became increasingly difficult.Source: §4.3 · p. 5
4.4 Possible Mitigations
- Suggested mitigations include whitelisting predefined websites, restricting online LLM access, and scanning executable prompts for malicious content and new variants.The proposed prompt-scanning capabilities include detecting prompts, distinguishing malicious from harmless prompts, and heuristic identification of variants.Source: §4.4 · p. 5
5 RELATED WORK
Related work covers LLM-assisted attack generation, phishing, and prompt injection, while distinguishing this paper’s focus from prior proxy-like scenarios. Earlier work described attacker-controlled web content and external APIs but did not show interaction with a user’s system.
5 RELATED WORK
- Prior studies examined LLM use to weaponize code, prompt injection, phishing, and inducing users to reveal confidential information or download malicious content.The passage cites studies [5] [7] [8] [16] on weaponized code and attacks, Gupta et al. [11] on jailbreaks and prompt injection, and phishing work.Source: §5 · p. 5
- Greshake et al. [9] described attacker-controlled web content influencing an LLM and an external API for communication back to the attacker.Their described prompts appeared on the attacker-controlled page.Source: §5 · p. 5
- The passage distinguishes that prior scenario by noting that its communication stayed within the LLM system and did not interact with the user’s system.Source: §5 · p. 5
6 CONCLUSION
The conclusion states that openly available LLMs and plugins can be misused for efficient, undetected attacks. The proof of concept uses plugins to connect an LLM proxy to C2 servers, motivating further mitigation strategies and security guidelines.
6 CONCLUSION
- The authors conclude that unsecured, openly available LLMs and plugins can be misused to perform efficient and undetected attacks.Source: §6 · p. 6
- The proof of concept demonstrates LLMs used as malware proxies through plugins to connect with command and control servers, enabling access to a victim’s machine without direct victim–LLM interaction.Source: §6 · p. 6
- The authors highlight the need for new mitigation strategies and further security guidelines for deploying LLMs.Source: §6 · p. 6