Source-linked AI summary
A Survey on Trustworthy LLM Agents: Threats and Countermeasures
Miao Yu, Fanci Meng, Xinyun Zhou, Shilong Wang, Junyuan Mao, Linsey Pang, Tianlong Chen, Kun Wang, Xinfeng Li, Yongfeng Zhang, Bo An, Qingsong Wen
TL;DR
LLM-based agents and MAS introduce trustworthiness issues that earlier LLM-focused research does not cover. TrustAgent surveys these systems through modular, technical, and multi-dimensional perspectives, identifying emerging threats, defenses, evaluations, and research gaps.
Problem
Existing trustworthiness surveys focused on LLMs or single agents do not fully address issues introduced by agents’ additional modules and MAS interactions.
Method
TrustAgent deconstructs agent and MAS components and organizes research by intrinsic and extrinsic modules, trustworthiness dimensions, and attack, defense, and evaluation techniques.
Results
The survey summarizes current attacks, defenses, and evaluations across agent modules and identifies technical gaps, including tool-invocation defenses and evaluations of memory and agent interactions.
Takeaways & Limitations
Trustworthy-agent research requires attention to the distinctive modules, interactions, and implications of agents and MAS.
Takeaways & Limitations
Current research lacks defense mechanisms for secure tool invocation, while trustworthiness evaluation of inter-agent interactions remains insufficient.
Abstract
from arXiv · showhide
With the rapid evolution of Large Language Models (LLMs), LLM-based agents and Multi-agent Systems (MAS) have significantly expanded the capabilities of LLM ecosystems. This evolution stems from empowering LLMs with additional modules such as memory, tools, environment, and even other agents. However, this advancement has also introduced more complex issues of trustworthiness, which previous research focused solely on LLMs could not cover. In this survey, we propose the TrustAgent framework, a comprehensive study on the trustworthiness of agents, characterized by modular taxonomy, multi-dimensional connotations, and technical implementation. By thoroughly investigating and summarizing newly emerged attacks, defenses, and evaluation methods for agents and MAS, we extend the concept of Trustworthy LLM to the emerging paradigm of Trustworthy Agent. In TrustAgent, we begin by deconstructing and introducing various components of the Agent and MAS. Then, we categorize their trustworthiness into intrinsic (brain, memory, and tool) and extrinsic (user, agent, and environment) aspects. Subsequently, we delineate the multifaceted meanings of trustworthiness and elaborate on the implementation techniques of existing research related to these internal and external modules. Finally, we present our insights and outlook on this domain, aiming to provide guidance for future endeavors.
1 Introduction
LLM-based agents and MAS extend LLMs with modules such as memory, tools, environments, and other agents, increasing capabilities while creating broader trustworthiness concerns. TrustAgent addresses this gap through a modular, technical, and multi-dimensional taxonomy of trustworthy agents.
- LLM-based agents add memory, tools, and environments to an LLM backbone, while MAS further introduce inter-agent communication.
- MAS > Single Agent > LLM summarizes the reported performance hierarchy, while added modules also expand trustworthiness risks.
- Additional modules expand attack surfaces and challenge existing defenses and evaluations developed for LLMs or single agents.
- TrustAgent organizes trustworthiness by internal and external agent components, covering brain, memory, tools, users, other agents, and environments.
- The framework also covers safety, privacy, truthfulness, fairness, and robustness across single-agent and MAS settings.
- Its technical perspective surveys implementation approaches for attacking, defending, and evaluating trustworthy agents, then identifies future research directions.
2 Intrinsic Trustworthiness
The survey treats intrinsic trustworthiness as a property of agents’ internal modules and analyzes those modules through their roles and technical trustworthiness paradigms.
- Intrinsic trustworthiness concerns the agent’s internal modules: its brain, memory, and tool-based behavior.
- The analysis first introduces each internal module’s role and functions, then summarizes attack, defense, and evaluation methods.
2.1 Trustworthy LLM as Brain
The agent brain is the core reasoning and decision-making module, but its richer information dynamics and connections create additional attack and evaluation challenges. The survey organizes brain threats, defenses, and evaluations while highlighting the need for collaborative security and dynamic assessment.
- Working mechanisms: The brain integrates auxiliary-module inputs and guides agent decisions, but those connections expand the attack surface beyond that of a single LLM.
- Attacks: Agent-brain attacks are organized into jailbreak, prompt injection, and backdoor paradigms based on their manipulation mechanisms.
- Attacks: Backdoor research includes infectious backdoors that implant one agent and influence other agents’ reasoning.
- Defenses: Defense approaches are organized by operational scope into alignment, single-model filters, and multi-agent shields.
- Evaluation: Brain evaluations include focused assessments for specific attacks or domains and general benchmarks spanning diverse domains and dimensions.
- Insight: Because compromised agents can spread attacks through MAS, the survey calls for mutual monitoring, distributed validation, and dynamic evaluation in evolving environments.
2.2 Trustworthy Memory in Retrieval
Agent memory comprises long-term retrieval stores and short-term interaction history, creating attack surfaces involving poisoning, privacy leakage, and misuse. The survey organizes corresponding attacks, defenses, and evaluation metrics while identifying weak cross-task generalization and limited benchmarking.
- Agent memory combines long-term vector-database retrieval with short-term dialogue or task history, enhancing capabilities while introducing trustworthiness risks.
- Attacks: Memory attacks are categorized as poisoning, privacy leakage, and misuse.Poisoning injects malicious long-term data; privacy leakage targets stored private information; misuse exploits multi-turn short-term memory.
- Attacks: Small amounts of injected malicious information can attack agents with high probability, while optimized retrieval conditions and backdoor triggers increase malicious-sample retrieval.
- Defenses: Defenses include query modification, output intervention, and detection or removal of harmful retrieved text.Examples include security-oriented query rewriting, isolate-then-aggregate response generation, and clustering retrieved embeddings to identify suspicious documents.
- Evaluation: Memory evaluation extends beyond attack success rate to retrieval metrics such as precision, recall, and F1-score.AgentPoison also uses retrieval attack success rate to measure malicious-trigger effects.
- Insights: Current memory attacks often lack cross-task generalization, and reliable systematic benchmarks for memory attacks and defenses remain needed.Future defenses target vector databases, privacy-preserving query or model changes, and multi-round adversarial dialogue training.
2.3 Trustworthy Tool for Action
Tools mediate agents’ information gathering and external actions, but planning, selection, execution, and tool capabilities introduce distinct attack interfaces. The survey reviews manipulation and abuse attacks, sparse defenses, and dataset- and sandbox-based evaluations.
- Tools connect agents to the external world through information gathering or action execution, including APIs, sensors, and embodied robots.
- Attacks: Tool-related threats comprise manipulation of invocation steps and abuse of tool capabilities against external entities.
- Attacks: Tool manipulation includes jailbreaks, prompt injection, tool injection, and command forgery that can induce unsafe behavior or privacy theft.ToolCommander injects malicious tools before manipulating selection with stolen queries; prompt injection can cause repetitive or irrelevant actions.
- Defenses: Defenses against tool-related attacks are notably scarce, with GuardAgent and AgentGuard offering API-based guardrails, workflow identification, and safety constraints.
- Evaluation: Dataset testing uses adversarial query collections, while sandbox simulation dynamically evaluates tool interactions and emergent multi-turn risks.ToolSword spans input, execution, and output attacks; ToolEmu simulates execution and uses an evaluator model to assess risks.
- Insights: Tool-enabled agents exhibit higher risks in multi-turn interactions, and future work must address malicious APIs and chains of multiple tool invocations.
3 Extrinsic Trustworthiness
Extrinsic trustworthiness concerns agents’ interactions with users, other agents, and environments. The survey examines these three interaction types through attacks, defenses, and evaluations, illustrated in Figure 5.
- External interactions are categorized as agent-to-agent, agent-to-environment, and agent-to-user relationships.
3.1 Agent-to-Agent Interaction
Agent-to-agent interaction enables cooperation, competition, and debate, but its propagative nature creates distinct trustworthiness risks across MAS. Existing work studies cooperative and infectious attacks, collaborative and topological defenses, and emerging evaluation methods.
- Agent-to-Agent Interaction: Agent-to-agent interactions span cooperation, competition, and debate, shaping system dynamics while introducing distinct trustworthiness risks.
- Attacks: Cooperative attacks manipulate inter-agent communication to spread harmful information, intercept coordination, or generate adversarial prompts.Examples include counterfactual-information propagation, intermediary interception, adversarial persuasion, and iterative adversarial-prompt refinement.
- Attacks: Infectious attacks propagate malicious effects across agents or MAS components, including text-based data theft, misinformation, and resource-draining self-propagation.
- Defenses: Collaborative defenses use information sharing, debate voting, adversarial techniques, and other cooperation patterns to analyze or filter agent responses.BlockAgents uses multi-round debate voting with Proofof-Thought consensus, while AutoDefense applies adversarial prompt filtering.
- Defenses: Topological defenses model MAS as networks and use topology optimization or graph neural networks to isolate threats and detect anomalous discourse.
- Evaluation and Insight: Agent-to-agent trustworthiness evaluation remains limited, with SafeAgentBench introducing a multi-agent execution environment and baseline models for assessment.The survey identifies a need for dedicated evaluation of inter-agent interactions and further anti-propagation methods.
3.2 Agent-to-Environment Interaction
Agent-to-environment interaction spans dynamic physical and digital settings requiring adaptive perception, reasoning, and action. Existing work is fragmented, while current evaluations and defenses do not yet systematically cover environmental interaction risks across domains.
- 3.2 Agent-to-Environment Interaction: Dynamic and heterogeneous environments create trustworthiness challenges because agent roles and environmental constraints influence perception, reasoning, and action.Examples include autonomous-driving errors and network disruptions across physical and digital domains.
- Physical Environments: Robotics, autonomous driving, and industrial-control studies use constraints, voting, attack detection, simulation, validation, reprompting, and code verification to improve safety or truthfulness.
- Digital Environments: Digital-environment research targets web navigation, healthcare diagnosis, financial tasks, and social-media regulation while addressing associated security, ethical, and compliance risks.
- 3.2.3 Insight: Current research often treats the environment as static and focuses on action outcomes, overlooking trustworthy agent-environment interaction as a vulnerability.Environmental feedback can be subtly altered to induce suboptimal or unsafe decisions.
- 3.2.3 Insight: Current evaluations mainly cover limited domain-specific safety scenarios and fail to address broader interdisciplinary and cross-domain challenges.
3.3 Agent-to-User Interaction
Trustworthy agent-to-user interaction requires attention to how users calibrate trust from agent behavior, alongside personalization, transparency, data protection, and multi-agent supervision. The survey frames adaptive calibration and explainability as directions for addressing these challenges.
- 3.3 Agent-to-User Interaction: Research emphasizes safety and reliability but understudies trust adjustment during interactions, especially how users respond to agent behavior.
- Trust Challenges: Personalization may increase engagement while creating manipulation risks, making transparency and clear explanations important for user risk management.
- Future Directions: Future work proposes adaptive trust calibration, optimized feedback, and unified regulation to support fairness and reliability across users and personalization settings.
- Future Directions: Explainable agents and supervisor agents are proposed to provide interpretable reasons and monitor multi-agent interactions for consistent responses.
4 Conclusion
The survey organizes trustworthy-agent research around six internal and external components and multiple trustworthiness dimensions, using attacks, defenses, and evaluations to synthesize the field. It identifies technical gaps and outlines future directions for agents and MAS.
- 4 Conclusion: The survey introduces six internal and external agent-system components and reviews trustworthy-agent research across dimensions including safety and privacy.It summarizes attack, defense, and evaluation perspectives and defines typical method paradigms for each.
- 4 Conclusion: The survey identifies gaps in trustworthy tool invocation defenses and trustworthiness evaluation for memory and agent interactions.
- 4 Conclusion: The authors highlight future research directions and insights concerning the distinctive attributes and implications of trustworthy agents and MAS.
A Trustworthiness Definition
TrustAgent defines agent trustworthiness across five core dimensions—safety, privacy, truthfulness, fairness, and robustness—while also recognizing accountability, ethics, transparency, and explainability. These dimensions address harmful behavior, data protection, reliable information, equitable treatment, and stable performance across agents and MAS.
- Core dimensions: Safety concerns preventing harmful actions or outputs, including threats amplified through adversarial inter-agent communication and system failures.Examples include cooperative attacks, infectious attacks, and agent-level jailbreaks.
- Core dimensions: Privacy protects user data and autonomy from unauthorized access or leakage through agents, memory, tools, and inter-agent communication.The passage identifies Prompt Infection, ToolCommander, memory poisoning, and embedding inversion as examples of privacy risks.
- Core dimensions: Truthfulness requires accurate, reliable, and consistent information while preventing misinformation propagation across agents and MAS.Propagation can arise from network dependencies, hallucinations, faulty tool outputs, or environmental misalignment.
- Core dimensions: Fairness requires impartial user treatment and equitable resource allocation without bias or discrimination.Risks include resource monopolization, biased task allocation, retrieval bias, and unequal tool access.
- Core dimensions: Robustness is the ability to maintain stable performance under diverse environments, uncertainty, and adversarial conditions.The passage highlights topological vulnerabilities, dynamic environmental changes, and tool-chain failures.
- Additional dimensions: Additional trustworthiness dimensions include accountability, ethics, transparency, and explainability.These address traceable actions, alignment with human values, visibility into internal processes, and clear justifications.
B Comprehensive Taxonomy
Figure 7 presents TrustAgent as a tree-structured taxonomy organized by agent modules and accompanied by cited references. It provides a comprehensive reference view of the framework’s categorized trustworthiness topics.
- References: The tree diagram includes the cited references associated with the taxonomy’s categories.The listed references include works on jailbreak, backdoor, and related attack categories.
- Taxonomy structure: Figure 7 organizes the comprehensive TrustAgent taxonomy according to agent modules.The figure is presented as a tree diagram for easy reference.