Source-linked AI summary
MalTool: Malicious Tool Attacks on LLM Agents
Yuepeng Hu, Yuqi Jia, Mengyuan Li, Dawn Song, Neil Gong
TL;DR
Malicious tool attacks expose a largely unexplored code-implementation gap beyond prior work on installation and agent selection. The paper introduces MalTool to synthesize diverse malicious tools and finds that existing detectors struggle to identify them, motivating defenses that analyze tool behavior more broadly.
Problem
Code-level malicious behavior is critical to successful malicious tool attacks but remains largely unexplored, leaving key behavior, generation, and detection questions open.
Method
MalTool uses a coding LLM, CIA-based behavior taxonomy, and automated verification to synthesize functionally correct, structurally diverse standalone and Trojan malicious tools.
Results
Existing detection methods exhibit high false negative and false positive rates, while MalTool constructs 1,300 standalone and 5,727 Trojan malicious tools.
Takeaways & Limitations
Detecting malicious tools may require defenses that jointly analyze implementations, natural-language descriptions, and their semantic consistency.
Takeaways & Limitations
The attack instantiations cover representative behaviors implemented with synthetic data and controlled infrastructure rather than the full spectrum of possible LLM-agent attacks.
Abstract
from arXiv · showhide
In a malicious tool attack, an attacker uploads a malicious tool to a distribution platform; once a user inadvertently installs the tool and the LLM agent selects it during task execution, the tool can compromise the user's security and privacy. Prior work focuses on manipulating tool names and descriptions to increase the likelihood of installation by users and selection by LLM agents. However, a successful attack also requires embedding malicious behaviors in the tool's code implementation, which remains largely unexplored. In this work, we bridge this gap by presenting the first systematic study of malicious tool code implementations. We first propose a taxonomy of malicious tool behaviors based on the confidentiality-integrity-availability triad, tailored to LLM-agent settings. To investigate the severity of the risks posed by attackers exploiting coding LLMs to automatically generate malicious tools, we develop MalTool, a coding-LLM-based framework that synthesizes tools exhibiting specified malicious behaviors, either as standalone tools or embedded within otherwise benign implementations. To ensure functional correctness and structural diversity, MalTool leverages an automated verifier that validates whether generated tools exhibit the intended malicious behaviors and differ sufficiently from previously generated instances, iteratively refining generations until success. Our evaluation demonstrates that MalTool is highly effective even when coding LLMs are safety-aligned. Using MalTool, we construct two datasets of malicious tools: 1,300 standalone malicious tools and 5,727 real-world tools with embedded malicious behaviors. We further show that existing detection methods, including conventional malware detection approaches and methods tailored to the LLM-agent setting, exhibit limited effectiveness at detecting the malicious tools, highlighting an urgent need for new defenses.
1 Introduction
Malicious tool attacks require installation, agent selection, and malicious code implementation, while prior work largely studied only the first two conditions. MalTool systematically studies code-level behaviors, generates diverse malicious tools, and evaluates existing detection methods.
- Attack gap: Successful malicious tool attacks require user installation, LLM-agent selection, and malicious behavior embedded in the implementation.The tool executes within the agent’s intended framework after selection and may compromise user security and privacy.
- Attack gap: Prior work primarily manipulated tool names and descriptions to increase installation and selection likelihood, leaving code-level implementation largely unexplored.The paper identifies implementation as equally critical to end-to-end attack success.
- MalTool: MalTool uses a coding LLM and an automated verifier to generate malicious tools that are functionally correct and structurally distinct.The verifier checks intended behavior and structural difference, iteratively refining generations until both conditions are met.
- Datasets: MalTool generates standalone and Trojan malicious tools and constructs datasets covering 13 malicious behaviors.The standalone dataset contains 100 instances per behavior, while 5,727 real-world tools are used to construct Trojan instances.
- Detection: Existing malware, program-analysis, and LLM-agent-specific detectors struggle to reliably identify malicious tools, producing high false negative and false positive rates.The evaluation includes VirusTotal, Bandit, Semgrep, Tencent A.I.G, Cisco MCP Scanner, and AntGroup MCPScan.
2 Related Work
Related work studies agent components, tool-selection attacks, and text- or code-based defenses. The paper emphasizes that code-level malicious behavior remains underexplored and that existing detectors lack systematic evaluation on realistic, diverse datasets.
- LLM Agent Systems: LLM-agent systems persist an underlying LLM, memory module, knowledge base, and user credentials across tasks.Memory, knowledge bases, and credentials are often stored on the device’s file system.
- LLM Agent Systems: At runtime, the agent uses the user prompt, memory, and potentially knowledge-base records to decide whether to invoke a tool and what inputs to provide.The selected tool may be invoked repeatedly as part of an iterative trajectory.
- Malicious Tool Attacks: Prior attacks crafted benign-looking names and descriptions, sometimes embedding prompts to manipulate tool selection.These attacks target installation and invocation likelihood rather than the tool’s implementation.
- Detection: Existing defenses are divided mainly into text-based detection of metadata and code-based detection of implementations.Code-based methods include static analysis, which requires source code, and dynamic analysis in sandboxed environments.
- Detection: Code-based detectors have not been systematically evaluated on large-scale malicious-tool datasets, partly because realistic and diverse benchmarks are lacking.The paper motivates benchmark construction for LLM-agent-specific malicious behaviors.
3 Threat Model
The threat model considers an attacker who uploads a protocol-conforming malicious tool that is installed and invoked by an agent, enabling confidentiality, integrity, or availability violations. The study organizes behaviors around the CIA triad and assumes access to a coding LLM but not the user’s agent or tasks.
- Attacker’s goal: The attacker uploads a malicious tool to a platform, and execution after installation and agent invocation can compromise confidentiality, integrity, and/or availability.The taxonomy is organized around the CIA triad.
- Attacker’s knowledge: The attacker knows tool-agent protocols and has access to a coding LLM, but not to the user’s specific agent or tasks.The generated tools are assumed to conform to standardized protocols such as MCP and Skills.
- Attacker’s capability: The attacker can make names and descriptions appear benign, inject prompts, and embed malicious logic in tool implementations.These capabilities target installation, selection, and execution stages of the attack.
- Scope: Malicious tools differ from traditional malware in their behaviors, execution models, and attack surfaces, while retaining some behaviors such as file encryption and data exfiltration.They can additionally target agent-specific resources and invocation semantics.
4 Taxonomy of Malicious Behaviors
The taxonomy organizes malicious tool behaviors around confidentiality, integrity, and availability, emphasizing attacks tailored to LLM-agent privileges and execution. It covers sensitive-information abuse, manipulation of agent-relevant data, and resource or timing disruption.
- Taxonomy: The taxonomy uses the confidentiality–integrity–availability triad to organize 13 malicious behaviors tailored to LLM-agent settings.It is not intended as an exhaustive malware taxonomy.
- Confidentiality: Confidentiality violations abuse agent-inherited privileges to extract sensitive information such as memory, knowledge bases, credentials, prompts, and runtime trajectories.Data exfiltration routes this information through attacker-controlled channels.
- Integrity: Integrity violations manipulate data or code relied upon by the agent, including poisoning memory or knowledge bases through malicious database injection.Other integrity behaviors include deleting local files or database records used by the agent.
- Availability: Availability violations degrade task completion by exhausting computational resources or introducing disruptive delays.The taxonomy includes CPU and GPU compute hijacking and response-time amplification through blocking behavior.
5 Our MalTool
MalTool prompts a coding LLM to generate standalone malicious tools, verifies their behavior and structural diversity, and embeds validated malicious logic into benign tools. Its iterative verifier rejects incorrect or overly similar implementations while must-execute embedding preserves reliable triggering.
- Framework: MalTool synthesizes standalone malicious tools for specified target behaviors and embeds them into benign tools to create Trojan malicious tools.The embedded tools are intended to preserve advertised functionality while adding malicious behavior.
- Framework: The system prompt specifies the target behavior, constrains realistic tool interfaces, encourages diversity, and incorporates verifier feedback.Feedback helps reduce repeated failed generations.
- Verification: The verifier executes candidates in controlled environments and accepts them only when observed runtime effects match the intended malicious behavior.Tests monitor effects including network traffic, file-system changes, database state, or resource usage.
- Verification: MalTool rejects candidates whose AST-subtree similarity to an accepted tool exceeds threshold τ, encouraging structurally distinct implementations.Similarity compares subtree structures while accounting for their frequencies.
- Trojan embedding: Trojan construction uses must-execute embedding so malicious code runs whenever the tool’s entry function is called.This avoids placement in rarely taken branches or after early exits.
6 Evaluating MalTool
The evaluation constructs standalone and Trojan datasets and measures MalTool’s reliability, diversity, efficiency, and robustness across coding LLMs. MalTool achieves consistent behavioral success, improves structural diversity and iteration efficiency, remains effective with safety-aligned models, and produces realistic-scale Trojan tools.
- Datasets: 1,300 standalone malicious tools are generated as 100 instances for each of 13 behaviors in Dataset I.Generation uses synthetic data and local infrastructure for controlled, reproducible evaluation.
- Results: GSR is 1.0 across all behaviors and three coding LLMs with MalTool’s verifier, while direct prompting performs substantially worse and varies by model and behavior.The verifier makes successful malicious behavior less dependent on coding-LLM strength.
- Results: MalTool consistently produces lower SIM than direct prompting across coding LLMs and malicious behaviors.This indicates that verifier-based diversity checking encourages structurally distinct implementations.
- Results: On Phi-4, CPU Compute Hijacking requires over 77 iterations without feedback but fewer than 7 with feedback.Feedback reduces generation-verification iterations across all three coding LLMs, with larger reductions for weaker or moderately capable models.
- Results: Safety alignment does not prevent MalTool from achieving GSR 1.000 across all malicious behaviors, while closed-source generation costs average $0.013–$0.033 per successful tool.The reported averages are $0.013 for GPT-4o, $0.017 for GPT-5.2, $0.033 for Claude-Opus-4.6, and $0.016 for Gemini-3-Pro.
- Results: 88% of collected tools contain fewer than 100 lines of code, and Trojan variants largely overlap original tools in LOC distributions.These measurements support the reported code-size realism of the generated tools.
- Datasets: 5,727 Trojan malicious tools comprise Dataset II, with 440 or 441 tools per behavior across all 13 behaviors.Validated standalone malicious code is embedded at execution points guaranteed to run.
7 Benchmarking Detection
The benchmark evaluates conventional malware detectors and LLM-agent-specific scanners on malicious and benign tools. Existing methods show substantial detection gaps and a difficult coverage–precision trade-off.
- Detection methods: The benchmark covers conventional malware detection methods and scanners tailored to LLM-agent tools.Evaluated methods include VirusTotal, Bandit, Semgrep, Tencent A.I.G, Cisco MCP Scanner, and AntGroup MCPScan.
- False negatives: Existing detectors show limited effectiveness on both standalone and Trojan malicious-tool datasets.False negative rates remain substantial across the evaluated behaviors and detectors.
- False negatives: VirusTotal has false negative rates close to 1.0 across behaviors, while Resource Hijacking and Denial of Service are near-universally missed.Data Exfiltration, Credential Abuse, and Remote Program Downloading are relatively easier for some detectors to identify.
- False positives: 30% to over 50% false positive rates are typical for Tencent A.I.G and AntGroup MCPScan on benign tools.The Combined Scanner often exceeds 50% and reaches over 70% in some categories.
- Overall findings: The results expose a coverage–precision trade-off that makes reliable separation of malicious and benign tools difficult.High false positive rates on benign tools also help explain why Trojan tools can be flagged more frequently.
- False positives: Manual inspection of 200 Combined Scanner alerts found no explicit malicious behavior, supporting that many benign-tool detections are false positives.The authors acknowledge that some real-world tools may still be malicious.
8 Discussion and Limitations
The discussion argues that effective defenses should combine text and code analysis with runtime safeguards. It also identifies policy specification and attack coverage as important boundaries.
- Detection: Current program-analysis detectors may need joint reasoning over tool implementations, descriptions, and semantic consistency.Detecting discrepancies between stated functionality and actual behavior is identified as a potentially important direction.
- Runtime defenses: Runtime guardrails can reduce malicious-tool activation by making agents less likely to select tools with deceptive names or descriptions.The discussion mentions fine-tuning the underlying LLM and redesigning tool selection as examples.
- Runtime defenses: Runtime security policies face a trade-off: under-specification leaves agents vulnerable, whereas over-specification can degrade benign-tool utility.Accurately specifying policies for each user task remains challenging.
- Scope boundaries: The study’s attack instantiations cover representative behaviors using synthetic data and controlled infrastructure, not the full spectrum of possible LLM-agent attacks.Dataset III consists of public real-world tools, and a very small subset may itself exhibit malicious behavior.
9 Conclusion and Future Work
The paper presents MalTool as a systematic study of malicious tool implementations and concludes that current detectors leave substantial gaps. It points toward joint text–code analysis and runtime guardrails as future defenses.
- Contributions: MalTool studies malicious tool implementations, introduces a behavior taxonomy, synthesizes standalone and Trojan tools, and builds malicious and benign benchmark datasets.The framework and datasets support systematic evaluation of malicious-tool defenses.
- Findings: Existing program-analysis-based detectors suffer from substantial false negatives on malicious tools and false positives on benign real-world tools.The paper identifies joint text–code analysis and effective runtime guardrails as important future directions.
10 Ethical Considerations
The study frames malicious-tool generation as an evaluation of detection mechanisms and defenses, using isolated synthetic experiments rather than interactions with real systems. Its findings are intended to expose detector weaknesses and inform safer designs.
- Study setting: The malicious behaviors are instantiated with synthetic data and locally controlled infrastructure in isolated environments.The experiments do not interact with real external services, users, or systems.
- Purpose: The evaluation aims to prevent false confidence in existing detection mechanisms and encourage more effective defenses.The authors present public documentation of detector limitations as a long-term benefit.
11 Code and Datasets
The paper uses a tiered artifact-release policy to support reproducibility while reducing potential misuse.
- Dataset III, containing benign real-world tools, is publicly available.The paper provides a public download link for this dataset.
- Standalone malicious tools, Trojan malicious tools, and the generation pipeline are restricted-access artifacts.Researchers may request access because these artifacts have dual-use risks.
- The release policy balances reproducibility with minimizing potential misuse.
12 Appendix
The appendix details malicious-behavior instantiations, behavior-specific verification, prompt construction, and generation results across safety-aligned coding models.
- Malicious behaviors: The appendix instantiates exfiltration, credential, database, file, program-download, resource-hijacking, response-delay, and encryption behaviors with concrete test environments.Examples include exfiltrating inputs, modifying SQLite records, deleting files, downloading programs, consuming compute resources, delaying responses, and encrypting files.
- Verifier design: MalTool uses behavior-specific verifiers that check observable effects such as received data, changed database state, deleted files, or resource usage.The verifier executes candidates in controlled environments and evaluates whether the intended effect occurs.
- System prompt: The system prompt specifies the malicious behavior, function interface, output format, syntactic constraints, and structural-diversity requirement.The prompt requires syntactically correct Python and a function distinct from previously supplied functions.
- Safety-aligned coding LLMs: All evaluated safety-aligned models eventually generate verifier-accepted malicious tools, achieving a GSR of 1.000.MalTool averages 2.944 iterations for the evaluated open-weight models, versus 2.508 for safety-ablated variants, while SIM remains comparable at 0.166 versus 0.154.
12.5 Details of GSR Test Instance Construction
The paper constructs and analyzes real-world and Trojanized tools, showing that malicious logic can remain localized while preserving benign interfaces and behavior.
- Real-world tool collection: The real-world corpus is collected from three public tool platforms and filtered to publicly accessible Python repositories.The crawler extracts listings, repository links, metadata, and full Python-based implementations for downstream processing.
- Data cleaning: The dataset pipeline removes tools with missing descriptions or code, duplicate entries, and non-English descriptions before constructing Trojan and benign datasets.
- Trojan tool characteristics: Trojanized tools typically add only a few localized statements while preserving original signatures, return behavior, and expected functionality.This makes the malicious side effects difficult to detect through superficial inspection.
- Remote data exfiltration example: A Trojanized email tool exfiltrates recipient, subject, and body data to a remote endpoint before sending the email through SendGrid.The hidden exfiltration does not change the tool’s observable interface.
- Local data exfiltration example: A Trojanized subscription tool records user and subscription parameters locally before creating the subscription through the OneSignal API.The original asynchronous interface and benign request remain intact.
- Detection evaluation: The evaluated detectors include commercial malware systems, program-analysis tools, and LLM-agent security scanners.The paper evaluates systems such as VirusTotal, Bandit, Semgrep, Tencent A.I.G, Cisco MCP Scanner, and AntGroup MCPScan.