Source-linked AI summary
Risks and Controls for Multi-Agent Systems: an analytical framework for deployment of AI agents across organisational boundaries
Alistair Reid, Simon O'Callaghan, Dustin Venini, Liam Carroll, Tiberio Caetano
TL;DR
AI agents increasingly interact across organisational boundaries, creating failures and governance challenges that individual organisations may not be able to control. This report introduces a three-tier framework for analysing risks, failure modes, controls, and who can act as interactions extend beyond internal deployments.
Problem
Increasing interactions among AI agents create safety and governance challenges, especially when failures cross organisational boundaries beyond any single organisation's controls.
Method
The report develops an analytical framework organised around three deployment tiers distinguished by the minimum common governance between interacting agents.
Results
The report shows that available controls and responsible actors depend on the deployment tier, with controls moving toward shared frameworks, public infrastructure, and collective action.
Takeaways & Limitations
Organisations, policymakers, and researchers can use the framework to reason about risks, controls, governance, and which actors are positioned to act across deployment settings.
Takeaways & Limitations
The report presents a selected repertoire of potential controls rather than an exhaustive set and leaves deployment organisations to choose among them based on context.
Abstract
from arXiv · showhide
This report presents a framework to help organisations, policymakers and researchers reason about the risks that emerge when AI agents interact with each other, how those risks change as interactions cross organisational boundaries, and the controls that may help address them. As organisations deploy AI agents, those agents will increasingly interact with each other: inside the organisation, with the agents of partners, customers and suppliers, and with unknown counterparties on the open internet. Failures can emerge from the interactions themselves, and once those interactions cross an organisation's perimeter, no single organisation can fully see, control or govern them. The report introduces three deployment tiers, defined by the minimum common governance binding any two interacting agents: singular governance, where one organisation governs every agent; federated governance, where multiple organisations deploy into a shared environment under agreed rules; and open environments, where agents operate with no central authority and shared standards are adopted voluntarily if at all. Within each tier, the report examines risk factors, failure modes and available controls. It identifies who is positioned to apply the controls, and where no actor is positioned to act, it characterises the gap and the collective action required to close it.
Copyright
The publication is generally released under a Creative Commons Attribution 4.0 International Licence, with specified exclusions and attribution requirements.
- The Commonwealth Coat of Arms, third-party content, trademarks, logos, and otherwise noted material are excluded from the Creative Commons licence.
- The publication may be copied, distributed, transmitted, adapted, and used commercially under the Creative Commons Attribution 4.0 International Licence, provided attribution is given.The full licence terms and a summary are available through the cited Creative Commons links.
- The publication should be attributed as Risks and Controls for Multi-Agent Systems: an analytical framework for deployment of AI agents across organisational boundaries.
Disclaimer
The report disclaims guarantees of accuracy, reliability, and completeness, places responsibility for independent inquiry on readers, and does not signal government commitment to a particular action.
- The authors state that AI tools assisted the report process, while the framework, analysis, conclusions, and final verification remain the authors’ responsibility.
- The Commonwealth does not guarantee the publication’s accuracy, reliability, or completeness and advises readers to conduct independent inquiries and obtain professional advice.
- The Commonwealth accepts no responsibility or liability for damage, loss, or expense resulting from reliance on the publication.
- The publication does not indicate Commonwealth commitment to a particular course of action.
Authors
The report was conceived, designed, and developed by Alistair Reid, Simon O’Callaghan, Dustin Venini, Liam Carroll, and Tiberio Caetano of the Gradient Institute.
- The report’s listed authors are Alistair Reid, Simon O’Callaghan, Dustin Venini, Liam Carroll, and Tiberio Caetano, affiliated with the Gradient Institute.
Executive summary
The report offers a framework for reasoning about risks, controls, and governance as AI-agent interactions cross organisational boundaries. It distinguishes three governance tiers and maps which actors can act, including coordination gaps requiring collective or systemic attention.
- AI agents increasingly interact across organisational boundaries, creating failures that may not be reachable by any single organisation’s controls.
- The report’s analytical framework helps practitioners, policymakers, and researchers reason about risks, controls, and governance beyond internal deployments.
- The framework distinguishes singular, federated, and open deployment tiers by the minimum common governance binding interacting agents.
- Controls and responsible actors change across tiers, moving beyond deploying organisations toward shared frameworks, public infrastructure, and collective action.
- For deploying organisations, the framework supports tier identification, control selection, deployment decisions, continuous governance, and recognition of where unilateral reach ends.
- For policymakers, the report identifies who can act on each risk and characterises gaps where no existing actor is positioned to respond.
- For researchers and standards bodies, it surfaces methodological and infrastructural gaps involving areas such as multi-agent evaluation and agent identity standards.
- The report aims to provide practitioners, policymakers, and researchers with a shared framework for addressing multi-agent safety at scale.
1 Introduction
AI-agent deployment is moving toward increasingly interconnected systems, including interactions across organisational boundaries where governance becomes more difficult. The report provides a three-tier analytical framework to examine risks, controls, coordination gaps and the limits of unilateral organisational reach.
- Organisations are deploying agents in production, while agents increasingly interact across business units, organisations and the open internet.
- Crossing organisational boundaries changes governance requirements because new failure modes and controls emerge beyond a single organisation’s unilateral reach.
- The report frames agent governance as an AI safety concern because agent failures can reduce opportunities for human intervention.
- Its analytical framework is organised around 3 deployment tiers defined by the minimum common governance binding interacting agents.
- The framework helps deploying organisations select tier-matched controls, policymakers locate coordination gaps, and researchers identify methodological and infrastructural gaps.
- The report focuses on LLM-based multi-agent interactions, assumes organisational governance for at least one interacting agent, and offers a selective rather than exhaustive catalogue.
2 Foundations
An LLM-based agent combines a language model with software components that enable an agentic loop.
- An LLM-based agent consists of an LLM taking instructions as input, together with a Harness and Scaffold enabling an Agentic loop.
2. Act on the environment using the tools and actions available
An agent acts through an iterative loop in which the model plans, the harness executes actions, and the system observes outcomes before repeating. Capability is improving rapidly, but performance remains uneven across domains and reliability remains important for deployment.
- The agentic loop distinguishes agents from workflows by enabling multi-step problem solving, strategy changes and adaptation to action outcomes.
- The model plans and reasons, the harness translates outputs into actions and outcomes into inputs, and the scaffold provides tools, modules and services.
- Increasingly sophisticated harnesses are eliciting greater capability from increasingly powerful models, particularly in software engineering.
- Software engineering is an exception because outputs are programmatically verifiable and errors surface quickly, whereas law and long-horizon planning lack immediate verification.
- LLM agents have jagged capability profiles and can be unreliable across repeated trials, affecting deployment appetite.
Single-agent guidance
Multi-agent systems arise when agents can affect one another through links over a shared substrate, with governance depending on the organisational boundaries those links cross. The report presents context-dependent control repertoires and describes an emerging ecosystem dominated by tightly scoped deployments.
- Multi-agent systems: A multi-agent system contains at least two agents whose linked behaviour can affect what other agents do or know.
- Multi-agent systems: Complex systems may include multiple agent instances, cross-organisational links and connections to the open web.
- Multi-agent systems: Links depend on a shared substrate such as a network, communication protocol or shared file system, alongside infrastructure for coordination and interoperation.
- Governance and controls: Governance varies by boundary: internal links are fully controllable, cross-organisational links may rely on contracts or platform terms, and open-web links may lack governance.
- Governance and controls: The report does not prescribe controls, instead offering a selective repertoire for organisations to weigh against risk exposure, implementation cost and use-case applicability.
- The agent ecosystem today: Current industry deployments are mostly tightly scoped, while flexible high-autonomy externally facing use cases remain unclear and startups may be early adopters.
3 Governing multi-agent deployments
Multi-agent governance requires a system-level approach because interactions introduce new failures that do not decompose across individual agents and can propagate or amplify existing failures.
- New failure types emerge from multi-agent interactions and require governance beyond single-agent concerns.
- Failures may not decompose across individual agents, making the interacting system—not isolated agents—the governance target.
- Interactions can propagate or amplify single-agent failures, strengthening the case for system-level governance.
- The report’s multi-agent focus does not mean single-agent failures are solved.
Analogy to human governance
Multi-agent governance combines familiar organisational controls with new demands created by machine-speed, scalable agent behaviour and the need for accountable responsibility.
- Analogy to human governance: Some multi-agent governance practices resemble human organisational controls, including least-privilege permissions, separated duties, audit trails, and counterparty due diligence.
- Analogy to human governance: Machine-speed failures can cascade faster than humans can intervene, saturating oversight regimes designed for human-paced escalation.
- Analogy to human governance: Scalable agent oversight therefore needs operational guardrails and controls that can operate at scale.
- Analogy to human governance: A responsible party should be identifiable and answerable for each agent’s actions, processes, and constraining guardrails.
- Analogy to human governance: These demands affect multiple organisational governance practices, with the report focusing on four key practices.
Attribution
Attribution becomes difficult in multi-agent systems because responsibility may need reconstruction across action chains, and some outcomes lack a single point of failure.
- Attribution assigns causal responsibility to agents and principals whose actions produced an outcome.
- In multi-agent settings, logging may require reconstructing the chain of actions after the fact to locate fault.
- Some system outcomes have no single point of failure because emergent behaviours are not exhibited by any agent in isolation.
Authorisation
Authorisation in multi-agent systems must govern not only an agent’s permissions but also the scope of authority delegated to newly created instances.
- Authorisation ensures that each agent acts with a mandate from a principal and within granted permissions.
- When agents can create other instances, delegated authority must generally be scoped to a subset of the parent agent’s permissions.
Oversight
Oversight requires maintaining visibility over agent activity so responsible humans can interpret decisions and intervene. In multi-agent systems, increasing action and communication volume can strain both visibility and legibility.
- Oversight keeps agent activity visible so responsible humans can interpret decisions and intervene.
- Single-agent oversight depends on reviewers keeping pace with decisions and understanding them.
- Multi-agent interactions increase action and communication volume with the number of agents, stressing oversight capacity and legibility.
Evaluation
The report treats multi-agent evaluation as an extension of single-agent evaluation, while governance complexity increases as interactions cross organisational boundaries. It analyses three tiers defined by their minimum common governance, with later tiers carrying earlier risks and requiring controls beyond one organisation’s reach.
- Multi-agent evaluation extends single-agent evaluation, which remains a difficult and unresolved problem.
- Governance complexity grows as agents move from bounded organisational settings toward open-ended interactions with unknown counterparties.
- The framework distinguishes three tiers by the minimum common governance binding any two interacting agents.
- Singular governance places every agent under one organisation, federated governance binds organisations through shared rules, and open environments lack central authority.
- Tier entry depends on deployment choices rather than technical capability, while risks and controls accumulate across successive tiers.
- The report examines risk factors, failure modes, controls, and unresolved problems beyond any single organisation’s reach across the three tiers.
4 Agents under singular governance
Singular governance covers linked-agent deployments in which one organisation has authority over the entire system, including internal help-desk, coding, personal, orchestration and incident-response agents. Its controls are actioned by that deploying actor, but human-facing interactions may introduce independently governed agents and higher-tier conditions.
- Singular governance describes linked agents governed by a single organisation with authority across the whole system.
- Examples include employee help-desk, coding, personal productivity, orchestrator-delegator, and autonomous incident-response agents.
- Telstra’s trial used agents to detect and resolve an infrastructure outage by shifting critical applications to healthy hardware.
- A customer’s or supplier’s agent can add an independently governed participant, shifting the system toward federated or open-environment conditions.
- Risk factors are deployment conditions that raise failure likelihood without themselves being failures, and their presence signals where controls warrant attention.
- The chapter selectively catalogues salient singular-governance risk factors and links them to enabled failure modes.
- Singular governance refers to system authority, even when an AI vendor jointly deploys or operates the agent.
Singular governance risk factors
Singular-governance systems can fail through miscommunication, cascading errors, conformity, drift, and context leakage even when individual agents appear safe. The section presents structured communication, verification, diversification, monitoring, and re-grounding as controls for these interaction-driven failures.
- Inter-agent communication failures arise when agents fail to converge on what was said, meant, done, or withheld.Roughly one in 3 multi-agent failures was attributed to inter-agent miscoordination, including reasoning-action mismatch, task derailment, clarification failures, and information withholding.
- A single upstream error can silently propagate through an agent system, with downstream agents correcting only a small fraction of initial mistakes.In a planner-executor-critic pipeline, the first flawed plan was the single largest predictor of failure.
- Cascade prevention combines workflow selection, system-level objective verification, model diversification, structured handoffs, and anomaly detection on intermediate outputs.These controls target unnecessary autonomy, shared-model monocultures, incomplete handoffs, and implausible intermediate results.
- Structured handoffs constrain incorrect-but-accepted outputs by enforcing schemas and enabling compliance checks, logging, and evaluation.Missing fields can force an agent to pause and request clarification before acting.
- False consensus can amplify through repeated interaction, while blind voting, varied contribution order, prompt mitigations, and bounded histories target conformity pathways.Hiding earlier answers and varying contribution order reduces anchoring on the first contribution.
- Persistent drift occurs when fabricated information becomes a shared convention, and current conformity benchmarks do not test whether such beliefs resist correction.External re-grounding, provenance on shared artefacts, and task-level verification are proposed responses.
- Frontier LLMs fail multi-user access-control tests, with privacy leakage worsening as conversations become longer.One study reported roughly 40% privacy-leakage rates on multi-party meeting summarisation despite explicit prompt instructions.