PaperlayerView exact revision on arXiv

Source-linked AI summary

TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems

Shaina Raza, Ranjan Sapkota, Manoj Karkee, Christos Emmanouilidis

arXiv:2506.04133v5cs.AI

TL;DR

Agentic Multi-Agent Systems create new trust, risk, security, and governance challenges through autonomous planning, coordination, memory, and tool use. This review synthesizes TRiSM principles for AMAS, proposes a unified framework, risk taxonomy, and evaluation metrics, and identifies controls and research directions. It concludes that AMAS require defense-in-depth safeguards, systematic evaluation, and regulation-aligned governance, while current evaluation and implementation practices remain constrained by scalability, ground-truth, trade-off, and integration challenges.

  • Problem

    Existing Agentic AI literature gives comparatively less attention to adversarial robustness, lifecycle governance, decision provenance, and system-level explainability in AMAS.

  • Method

    The review synthesizes TRiSM principles for AMAS through a unified five-pillar framework, an AMAS-specific risk taxonomy, evaluation metrics, control mappings, and a research roadmap.

  • Results

    The review identifies AMAS-specific risks and recommends defense-in-depth controls spanning prompt hygiene, memory sanitization, orchestrator authentication, tool sandboxing, continuous monitoring, encryption, access control, and privacy-preserving computation.

  • Takeaways & Limitations

    Responsible AMAS development should align explainability, ModelOps, security, privacy, lifecycle governance, evaluation, and regulatory compliance at the system level.

  • Takeaways & Limitations

    AMAS evaluation remains resource-intensive, often lacks ground truth for emergent coordination, involves domain-dependent metric trade-offs, and faces costly integration and monitoring challenges.

Abstract

from arXiv · show

Agentic AI systems, built upon large language models (LLMs) and deployed in multi-agent configurations, are redefining intelligence, autonomy, collaboration, and decision-making across enterprise and societal domains. This review presents a structured analysis of Trust, Risk, and Security Management (TRiSM) in the context of LLM-based Agentic Multi-Agent Systems (AMAS). We begin by examining the conceptual foundations of Agentic AI and highlight its architectural distinctions from traditional AI agents. We then adapt and extend the AI TRiSM framework for Agentic AI, structured around key pillars: \textit{ Explainability, ModelOps, Security, Privacy} and \textit{their Lifecycle Governance}, each contextualized to the challenges of AMAS. A risk taxonomy is proposed to capture the unique threats and vulnerabilities of Agentic AI, ranging from coordination failures to prompt-based adversarial manipulation. To support practical assessment in Agentic AI works, we introduce two novel metrics: the Component Synergy Score (CSS), which quantifies the quality of inter-agent collaboration, and the Tool Utilization Efficacy (TUE), which evaluates the efficiency of tool use within agent workflows. We further discuss strategies for improving explainability in Agentic AI, as well as approaches to enhancing security and privacy through encryption, adversarial robustness, and regulatory compliance. The review concludes with a research roadmap for the responsible development and deployment of Agentic AI, highlighting key directions to align emerging systems with TRiSM principles-ensuring safety, transparency, and accountability in their operation.

1. Introduction

Agentic Multi-Agent Systems (AMAS) combine LLM-powered planning, tool use, memory, and coordination, creating system-level risks that require an integrated TRiSM perspective. This review addresses that gap through a structured literature review, a unified framework, an AMAS risk taxonomy, evaluation metrics, and a research roadmap.

  • Agentic AI and AMAS: AMAS coordinate specialized LLM-powered agents that plan, use tools, retain memory, and adapt workflows through interactions with users and other agents.The paper distinguishes these systems from earlier task-specific, largely deterministic agents.
  • Risk Motivation: AMAS introduce cascading errors, biased decisions, unintended interactions, prompt-level attacks, memory poisoning, impersonation, and privacy leakage.Planning, coordination, external tool calls, and collaborative settings amplify these risks.
  • Research Landscape: Agentic AI research accelerated sharply, with multi-agent publications rising from 890 in 2019 to 18,500 in 2024 and LLM-based agent papers reaching 9,800 after ChatGPT’s release.The review also reports journal articles comprising 61.2% of the publication landscape and rising public search interest by late 2024.
  • Research Gap: The literature comparatively underemphasizes adversarial robustness, lifecycle governance, decision provenance, and system-level explainability for multi-agent deployments.Prior surveys typically emphasize architectures, capabilities, communication, and applications rather than AMAS-specific dynamics.
  • Contributions: The proposed contribution combines five TRiSM pillars, an AMAS-specific risk taxonomy, CSS and TUE metrics, technique mapping, and a roadmap for scalable, verifiable, regulation-aligned systems.The framework covers Explainability, ModelOps, Security, Privacy, and lifecycle Governance; CSS measures collaboration quality and TUE assesses tool-call correctness and efficiency.
  • Review Method: The review uses a structured literature-review process covering major digital libraries, publications from 2022–2025, foundational earlier work, and two-stage screening.The review is guided by questions concerning AMAS-specific risks, TRiSM controls, and evaluation gaps.

2. Fundamentals of Agentic AI Systems

Agentic AI systems use LLMs to coordinate roles, memory, planning, tools, and environmental interaction for adaptive, long-horizon task execution. AMAS architectures connect these capabilities through communication, orchestration, shared memory, oversight, security, privacy, and monitoring layers.

  • Architectural distinction: Agentic AI systems extend traditional rule-based and cognitive agents with LLM-supported coordination, persistent context, dynamic role orchestration, and tool use.These capabilities support long-horizon execution and cross-domain generalization beyond reactive pipelines.
  • Agent coordination: AMAS coordinate multiple specialized LLM agents to decompose complex goals, assign roles, exchange messages, and adapt workflows through tools, users, and other agents.The central controller iterates through planning, acting, and context updates based on tool results and feedback.
  • Memory and data: Memory preserves context across steps and sessions, but shared stores and retrieval introduce stale-context, poisoning, leakage, and accountability risks.Short-term memory remains in working context, whereas long-term memory commonly uses embedding-based retrieval.
  • Tools and execution: Tool-use interfaces let agents issue structured commands to external systems, including web search, APIs, code execution, and databases.Returned tool outputs become observations for subsequent reasoning, enabling capabilities beyond text generation.
  • System architecture: AMAS combine perception, reasoning, action, learning, communication, storage, and retrieval into an integrated architecture.Perception converts environmental inputs into model-readable representations, while actions produce new observations for iterative interaction.
  • Governance and control: Monitoring and governance collect logs, traces, and metrics while enforcing policies and enabling authentication, auditability, intervention, and lifecycle oversight.Protocolized communication and monitoring create practical control points, although the closed-loop architecture expands the attack surface and complicates accountability.

3. Threats and Risks in Agentic AI

The review organizes AMAS threats into adversarial attacks, data leakage, agent collusion, and emergent behaviors. Autonomy, shared memory, orchestration, and tool use can allow failures to cascade across agents and persist through memory.

  • Risk taxonomy: AMAS risks are grouped into four classes: adversarial attacks, data leakage, agent collusion, and emergent behaviors.The taxonomy focuses on system-level risks arising from multi-agent interaction.
  • Adversarial attacks: Prompt injection and engineered reasoning traps can cascade across agents, including role-swapping attacks that exploit inter-agent dependencies.These attacks manipulate agent interactions rather than only individual model inputs.
  • Data leakage: Shared memory and inter-agent communication can expose sensitive information when boundaries, sanitization, or access controls are weak.Persistent memory can also retain and propagate injected or low-quality content without strong versioning and audit controls.
  • Agent collusion: Collaboration mechanisms can produce groupthink or collusive failure when agents reinforce flawed assumptions without external checks or objective feedback.ChatDev-like workflows illustrate how shared planning errors may propagate through mutual validation.
  • Emergent behaviors: Interactions among agents, tools, and memory can generate unpredictable strategies and emergent behaviors that evade conventional test suites.These behaviors are a distinct risk class in the review’s taxonomy.
  • Mitigation direction: The review motivates defense-in-depth controls spanning prompt hygiene, memory sanitization, orchestrator authentication, tool sandboxing, and continuous monitoring.These controls address cascades, persistent memory failures, and tool-mediated risks across the system.

4. TRiSM Framework for Agentic Multi-Agent Systems

The review adapts TRiSM to AMAS through five pillars—Explainability, ModelOps, Application Security, Model Privacy, and Governance—aligned with system architecture and external frameworks. It emphasizes distributed decision tracing, lifecycle controls, defense-in-depth, privacy-preserving techniques, and human oversight.

  • Framework structure: The AMAS TRiSM framework comprises Explainability, ModelOps, Application Security, Model Privacy, and Lifecycle Governance.These pillars are treated as distinct risk domains and mapped onto core architectural layers.
  • Explainability: AMAS explainability must trace individual agent decisions and inter-agent dynamics that produce emergent system-level behavior.Relevant approaches include provenance graphs, role-based interpretability, attribution, counterfactual analysis, and reasoning-trace logging.
  • Explainability: Intermediate-step verification is necessary because vanilla CoT can produce plausible but unverified reasoning, allowing errors to propagate and compound.The framework therefore emphasizes verification and traceability across distributed decisions.
  • Lifecycle Governance: Governance aligns lifecycle management, security, privacy, accountability, transparency, and human oversight with frameworks including the EU AI Act, NIST AI RMF, and ISO/IEC 42001.Governance structures include documented processes, auditability, policy constraints, and intervention mechanisms.
  • ModelOps: ModelOps extends lifecycle management through hierarchical monitoring, prompt versioning, CI/CD safety gates, and pre-deployment simulation.These controls support governed updates and monitoring across agent workflows.
  • Application Security: Application security emphasizes Plan-Then-Execute, least-privilege access control, cross-agent verification, authentication, and tool sandboxing.The security pillar addresses agent coordination and external tool integrations through defense-in-depth controls.
  • Model Privacy: Privacy preservation uses differential privacy, secure multi-party computation, homomorphic encryption, trusted execution environments, and controlled data access.These mechanisms protect sensitive information during inter-agent communication and tool interaction.

5. TRiSM Evaluation of Agentic AI Systems

The evaluation framework treats Agentic AI as multidimensional, combining trustworthiness, explainability, user-centered performance, coordination, and composite reporting. It introduces CSS and TUE to measure inter-agent enablement and tool-use quality, while recommending metric vectors and threshold or Pareto-based selection.

  • Evaluation dimensions: Evaluation covers trustworthiness, explainability, user-centered performance, coordination, and composite reporting because accuracy alone misses coordination, tool-use, and autonomy failures.The framework includes robustness, violation rates, calibration, explanation quality, user satisfaction, goal fulfillment, and coordination signals.
  • Trustworthiness: Trustworthiness combines calibrated task success, robustness under shifts, and alignment or safety compliance, with a penalty for normalized violations.A, R, and L are normalized to [0, 1], while V is a nonnegative violation or self-serving penalty; the weights sum to one and λ controls penalty strength.
  • Coordination: Coordination measures shared plans, redundant work, dependency resolution, communication efficiency, consistency, and deadlock or conflict rates.The framework adds CSS for enabling contributions across agents and TUE for the correctness and efficiency of tool calls.
  • Coordination metrics: CSS quantifies how effectively one agent’s actions improve a peer’s downstream performance, combining pairwise impact with the recipient’s conditioned output quality.For the illustrative Planner–Coder–Tester link, Impact = 0.35 and Quality = 0.92 produce CSS = 0.35 × 0.92 ≈ 0.32.
  • Tool-use metrics: TUE evaluates tool invocation through selection, argument validity, execution success, outcome correctness, and efficiency.The linear form weights selection, arguments, execution, outcomes, and efficiency; efficiency is the normalized inverse of latency, token, and call cost.
  • Composite reporting: Model selection should report the full metric vector and use minimum thresholds or Pareto-optimality rather than relying only on a weighted scalar.Scalarization can hide critical single-dimension failures, so thresholding and non-dominated-model analysis provide additional safeguards.

6. Security and Privacy in Agentic AI Systems

Security and privacy in AMAS require layered, system-level controls spanning data, execution, communication, model robustness, and runtime behavior. The section combines encryption, least-privilege access, adversarial defenses, monitoring, privacy-preserving computation, and governance evidence.

  • Security Architecture: AMAS require layered defenses because multi-agent collaboration expands the attack surface across data, execution, communication, and model robustness.Recommended controls include encryption, access control, adversarial defenses, and runtime monitoring tailored to multi-agent settings.
  • Security Controls: Encryption protects inter-agent exchanges through SSL/TLS, homomorphic encryption, and secure enclaves.These mechanisms target confidentiality when agents exchange prompts, models, or partially processed results.
  • Security Controls: Least-privilege access control limits privilege escalation and unauthorized tool invocation for specialized agent roles.RBAC/ABAC policies can restrict orchestrators, shared memory, and role-specific capabilities.
  • Security Controls: Adversarial training, prompt and tool-output validation, and runtime monitoring address injection, poisoned outputs, malformed results, and anomalous behavior.Monitoring can combine auditing, anomaly detectors, and trust scoring among agents.
  • Privacy: Privacy protection combines differential privacy, data minimization, anonymization, memory controls, and secure computation.SMPC, homomorphic encryption, and TEEs support joint computation over protected data, while temporary buffers and redaction reduce unnecessary retention.
  • Evaluation: The proposed Composite Security Score combines input validation coverage, tool access control, memory integrity, and inter-agent authentication strength.Defense Effectiveness Rate measures the proportion of attacks detected and mitigated, while false-positive and false-negative rates capture detection errors.

7. Compliance with AI Governance and Policy Frameworks for Agentic AI Systems

Compliance for agentic AI is presented as a lifecycle program combining standards, regulation, secure development, auditability, and runtime enforcement. The framework links technical controls to evidence needed for governance and incident response.

  • Governance Frameworks: NIST AI RMF, ISO/IEC standards, the EU AI Act, and sectoral laws provide complementary governance requirements for agentic systems.Relevant obligations include risk management, transparency, data governance, human oversight, minimization, purpose limitation, and consent.
  • Auditability: Comprehensive logging and decision provenance enable auditors to reconstruct prompts, plans, actions, tool calls, hand-offs, and approvals.Records should preserve timestamps, agent identities, rationales where feasible, and hashes or version identifiers for models, tools, and data.
  • Runtime Enforcement: Policy enforcement keeps agent behavior within allowed bounds through orchestrator controls, scoped memory, least-privilege access, sandboxing, and I/O mediation.Sensitive operations can require human approval, while TTLs, redaction, environment scoping, and egress controls constrain execution.
  • Evidence and Documentation: Compliance requires technical documentation, model and data cards, evaluation reports, lineage, licenses, known hazards, and versioned artifacts.Documentation supports audit and regulator expectations by recording purpose, risks, limitations, misuse, oversight, and residual risk.
  • Lifecycle Governance: Agentic compliance is a lifecycle program coupling standards, regulation, secure development, and runtime enforcement rather than a single checklist.This combination creates evidenceable trust and reduces the blast radius of inevitable failures.

8. Discussion

The discussion frames TRiSM as a practical foundation for oversight, evaluation, compliance, and future research, while emphasizing substantial scalability, measurement, and validation limits. It recommends stronger benchmarks, evolving defenses, human oversight interfaces, and multidisciplinary governance.

  • Oversight and Accountability: TRiSM uses continuous oversight guardrails and guardian agents to filter sensitive data, establish behavioral baselines, and enforce runtime policies.The discussion also warns against excessive agency and supports defined human oversight roles.
  • Evaluation: The evaluation framework separates trustworthiness, explainability, user-centered performance, and coordination metrics instead of relying on monolithic accuracy.CSS and TUE target multi-agent phenomena that single-agent metrics cannot represent.
  • Critical Limitations: Evaluation remains limited by costly human-centered assessment, missing ground truth for emergent coordination, metric trade-offs, drift, subjective measures, and score gaming.Composite weights vary by domain, and optimizing a single score may invoke Goodhart’s Law.
  • Implementation Constraints: Production deployment can add latency, computational overhead, infrastructure complexity, staffing costs, and alert volumes that overwhelm security teams.The discussion presents phased, risk-driven adoption as a more feasible operational strategy.
  • Regulatory Alignment: TRiSM practices converge with emerging regulation by translating safeguards into compliance-ready processes and evidence.The discussion specifically connects TRiSM with risk management, transparency, data governance, and human oversight requirements.
  • Research Gaps: No consensus exists on standardized TRiSM benchmarks, while many controls lack real-world validation and adversarial learning remains challenged by adapting attackers.Proposed priorities include stress testing, longitudinal drift evaluation, and deployment case studies reporting effectiveness and operational costs.
  • Future Roadmap: Future work should extend TRiSM to multimodal and embodied agents, open governance benchmarks, stronger adversarial testing, human oversight interfaces, and regulatory sandboxes.The roadmap also proposes multidisciplinary governance boards including ethicists, legal experts, domain specialists, and community representatives.

9. Conclusion

The review frames AMAS as powerful but expanded-risk systems requiring a TRiSM-oriented approach. It proposes a five-pillar framework, AMAS-specific risk taxonomy, practical evaluation metrics, technical safeguards, and a roadmap centered on robustness, governance, and standardized evaluation.

  • Framework Contribution: The review organizes TRiSM for LLM-based AMAS around Explainability, ModelOps, Security, Privacy, and lifecycle Governance.The framework is tailored to how multi-agent systems collaborate, plan, use tools, retain memory, and make decisions.
  • Risk Taxonomy: The risk taxonomy captures prompt injection, memory poisoning, collusive failures, cascading failures, and other AMAS-specific threats.These risks reflect the expanded risk surface created by autonomous collaboration and coordination.
  • Evaluation Metrics: CSS and TUE evaluate inter-agent enablement and the correctness and efficiency of tool use.The metrics address coordination and workflow properties not captured by conventional single-agent evaluation alone.
  • Research Roadmap: The conclusion identifies stronger adversarial robustness, clearer governance and audit protocols, and standardized benchmarks as priorities for responsible deployment.These directions target safety, transparency, accountability, trustworthiness, and coordination under realistic conditions.

Appendix

The appendix provides a figure comparing traditional AI agents with Agentic AI and tables listing terminology and abbreviations used in the paper.

  • Table A.1 lists abbreviations used in the paper.
  • Figure A.1 compares a traditional AI agent with Agentic AI.
  • Table A.2 lists key terminologies for LLM-based agentic AI systems.
Loading 2506.04133v5…