PaperlayerView exact revision on arXiv

Source-linked AI summary

Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agents

Md Shamsujjoha, Qinghua Lu, Dehai Zhao, Liming Zhu

arXiv:2408.02205v4cs.SEcs.AI

TL;DR

Runtime guardrail design for FM-based agents lacks comprehensive guidance, despite challenges from agent autonomy, non-determinism, and multiple pipeline stages and artifacts. The paper reviews selected studies to build a taxonomy and a Swiss Cheese-inspired multi-layered reference architecture. It identifies guardrail qualities and design options and outlines an architecture spanning quality attributes, pipelines, and artifacts.

  • Problem

    Comprehensive studies and design guidance for runtime guardrails for FM-based agents are lacking, despite their autonomous and non-deterministic behavior.

    Source: §I · p. 1, §II · p. 2

  • Method

    The paper synthesizes 32 selected studies through a systematic literature review to develop a guardrail taxonomy and multi-layered reference architecture.

    Source: §I · p. 1, C. Study Search and Filtering Process · p. 3

  • Results

    The paper identifies 14 qualities and design options for runtime guardrails, then proposes an architecture spanning quality attributes, pipelines, and agent artifacts.

    Source: §I · p. 1, §VII · p. 10

  • Takeaways & Limitations

    The taxonomy and architecture provide guidance for designing runtime guardrails from a software-architecture perspective.

    Source: §I · p. 1

    Guardrail generalizability across FM-based agent contexts remains limited, and rapid evolution may make findings outdated.

    Source: §VI · p. 10

Abstract

from arXiv · show

Foundation Model (FM)-based agents are revolutionizing application development across various domains. However, their rapidly growing capabilities and autonomy have raised significant concerns about AI safety. Researchers are exploring better ways to design guardrails to ensure that the runtime behavior of FM-based agents remains within specific boundaries. Nevertheless, designing effective runtime guardrails is challenging due to the agents' autonomous and non-deterministic behavior. The involvement of multiple pipeline stages and agent artifacts, such as goals, plans, tools, at runtime further complicates these issues. Addressing these challenges at runtime requires multi-layered guardrails that operate effectively at various levels of the agent architecture. Therefore, in this paper, based on the results of a systematic literature review, we present a comprehensive taxonomy of runtime guardrails for FM-based agents to identify the key quality attributes for guardrails and design dimensions. Inspired by the Swiss Cheese Model, we also propose a reference architecture for designing multi-layered runtime guardrails for FM-based agents, which includes three dimensions: quality attributes, pipelines, and artifacts. The proposed taxonomy and reference architecture provide concrete and robust guidance for researchers and practitioners to build AI-safety-by-design from a software architecture perspective.

I. INTRODUCTION

FM-based agents’ autonomy and non-determinism raise safety concerns, while existing guardrails often focus on single layers and narrow agent artifacts. The paper synthesizes prior research into a taxonomy and proposes a Swiss Cheese-inspired architecture for multi-layered runtime guardrails.

I. INTRODUCTION

  • FM-based agents perceive context, reason, plan, and execute workflows through interactions with models, tools, knowledge bases, and other agents.The paper describes foundation models as versatile pretrained models that can adapt to downstream tasks, while noting that complex tasks may require detailed instructions.Source: §I · p. 1
  • The agents’ autonomous and non-deterministic behavior raises safety concerns, including harmful or offensive content, dangerous outcomes, and misinformation.Runtime guardrails are mechanisms integrated into an agent architecture to safeguard behavior during operation.Source: §I · p. 1
  • Single-layer guardrails focused on narrow artifacts can overlook qualities such as customizability and interpretability, leaving risks able to bypass a failed layer.The paper argues that multiple layers can provide a more robust defense.Source: §I · p. 1
  • The paper proposes a taxonomy based on a systematic literature review and a reference architecture organized around quality attributes, pipelines, and agent artifacts.Layers can target qualities such as privacy and security, pipeline stages such as prompts and results, and artifacts such as goals, plans, and tools.Source: §I · p. 1

II. BACKGROUND AND RELATED WORK

  • Prior work discusses foundation models, FM-based agents, and responsible AI, while comprehensive studies of runtime guardrails for agents remain lacking.Related work includes reviews of pretrained models and taxonomies of FM-based systems and responsible-AI considerations.Source: §II · p. 2, A. Recent State-of-the-Art Works on… · p. 2

B. Guardrails Approaches and Tools for FM-Based Agents

Prior runtime guardrail efforts include monitoring, filtering, and adaptive controls, but comprehensive design guidance for multi-layered guardrails remains absent. The study uses a literature-review protocol to examine guardrail qualities, design options, and an architecture for FM-based agents.

B. Guardrails Approaches and Tools for FM-Based Agents

  • Existing approaches include input and output monitoring or filtering, agent oversight, continuous validation, adaptive fail-safes, and dynamic access controls, but no framework provides comprehensive multi-layered design guidance.Runtime guardrails differ from design-time alignment because they monitor and control behavior during operation.Source: B. Guardrails Approaches and Tools for… · p. 2

III. METHODOLOGY

  • The study follows Kitchenham’s guidelines and uses PICOC to define its scope and develop a research protocol.The protocol specifies scientific resources, search terms, checklists, and study inclusion and exclusion criteria.Source: §III · p. 2, A. Research Approach · p. 2

B. Research Questions

  • The study asks which qualities and design options matter for runtime guardrails and how multi-layered guardrails can address FM-based agents’ distinctive challenges.The third question examines adapting the Swiss Cheese Model across agent artifacts.Source: B. Research Questions · p. 3

C. Study Search and Filtering Process

The review searched and filtered studies, extracted and assessed their evidence, and used the selected studies to develop a taxonomy. The taxonomy distinguishes guardrail quality attributes from practical design dimensions.

C. Study Search and Filtering Process

  • 32 selected studies supplied the findings used to develop the taxonomy after searching, screening, manual searching, and backward snowballing.The search and filtering process was conducted in March 2024, so later-published studies were not investigated.Source: C. Study Search and Filtering Process · p. 3

D. Data Extraction and Quality Assessment

  • Data extraction captured guardrail definitions, quality attributes, design options, and contextual information; studies were rated against five quality criteria.Studies averaging below 2 on the 1-to-5 assessment scale were excluded from further analysis.Source: D. Data Extraction and Quality… · p. 3

IV. TAXONOMY OF GUARDRAILS FOR FM-BASED AGENTS

  • The taxonomy synthesizes 32 studies through iterative clustering and validation, organizing findings into quality attributes and design dimensions.It also standardizes terminology and refines overlapping attributes, including attributes relevant to both developers and stakeholders.Source: C. Study Search and Filtering Process · p. 3, §IV · p. 4

A. Quality Attributes of Guardrails

  • The quality attributes are systematically synthesized and refined to represent distinct contributions relevant to guardrail performance, security, reliability, trust, and system outcomes.The paper discusses these attributes in a dedicated subsection.Source: A. Quality Attributes of Guardrails · p. 4

1) Accuracy:

The section describes accuracy, efficiency, privacy, security, and safety as relevant concerns for FM-based agents and runtime guardrails. It connects these concerns to errors, resource use, sensitive data, malicious activity, and harmful or misleading outputs.

1) Accuracy:

  • Accuracy guardrails address hallucinations, misinformation, and disinformation, since errors can mislead users, damage credibility, and lead to financial or legal costs.The paper notes that companies use guardrails to label AI-generated content to prevent deepfakes and misinformation.Source: 1) Accuracy · p. 4

2) Efficiency:

  • Guardrails can dynamically manage resources and filter irrelevant inputs to reduce resource-intensive tasks and endless loops that slow agent responses.The paper frames efficiency around users’ expectations for fast responses.Source: 2) Efficiency · p. 4

3) Privacy:

  • Privacy risks include leakage of sensitive data through direct responses, statistical inference, or inadvertent model-output disclosures.The passage cites a 2023 incident in which Samsung employees leaked proprietary information into ChatGPT, after which Samsung banned it.Source: 3) Privacy · p. 4

4) Security:

  • Security guardrails address malicious activity, data manipulation, confidentiality breaches, adversarial queries, and threats to integrity, availability, and performance.The passage describes risks from both hacking and misuse by third-party providers, including when access is authorized.Source: 4) Security · pp. 4–5

5) Safety:

  • Safety concerns include inappropriate, offensive, incorrect, or questionable outputs, which can have severe consequences in medical diagnosis or self-driving contexts.Such outputs can also damage an agent’s credibility and acceptance.Source: 5) Safety · p. 5

6) Fairness:

The section characterizes fairness and compliance concerns and describes guardrail qualities spanning generalizability, customizability, adaptability, traceability, and portability.

6) Fairness:

  • Fairness risks include biased agent outputs arising from training data, model algorithms, or deployment context, such as demographic preferences in recruitment screening.Such bias can affect credibility.Source: 6) Fairness · p. 5

7) Compliance:

  • Compliance guardrails monitor operations, automate checks, and support audits against legal, regulatory, data-protection, and licensing requirements.Real-time detection and restriction can mitigate unauthorized use of generated content; watermarking and labeling support ownership and licensing compliance.Source: 7) Compliance · p. 5

8) Generalizability:

  • Generalizability means guardrails work in real time across multiple layers and diverse scenarios without prior configuration.The attribute also covers extending to new domains and unexpected inputs without significant reconfiguration or performance degradation.Source: 8) Generalizability · p. 5

9) Customizability:

  • Customizable guardrails provide tailored, layer-specific protection aligned with operational goals, data characteristics, and regulatory environments.For example, a customer-service chatbot could prioritize guardrails and adjust data handling based on user location.Source: 9) Customizability · p. 5

10) Adaptability:

  • Adaptability is the ability to adjust and remain effective as conditions and data landscapes evolve, including by responding to emerging threats without manual reconfiguration.An example is automatically updating a chatbot’s guardrails to detect and block new offensive terms.Source: 10) Adaptability · p. 5

11) Traceability:

  • Traceability records guardrail-relevant origins, processes, and decision paths, creating an auditable trail that can support troubleshooting, accountability, and compliance.Records can include FM inputs and outputs, external tools, data sources, and model modifications.Source: 11) Traceability · p. 5

12) Portability:

  • Portability means guardrails can be adapted across different FM-based agents, architectures, and environments with minimal adjustments.The same guardrail may, for example, support content moderation in both a customer-service chatbot and a social-media platform.Source: 12) Portability · p. 5

13) Interoperability:

This section describes interoperability and interpretability as guardrail qualities, then organizes design options around actions and their application.

13) Interoperability:

  • Interoperable guardrails work across agents and technologies, interface with diverse components, services, APIs, and data formats, and support consistent security, privacy, and compliance protocols.The described benefits include less complexity in maintaining disparate protections and support for collaborative data sharing.Source: 13) Interoperability · p. 6

14) Interpretability:

  • Interpretability is the clarity and transparency of guardrail operation, helping users and stakeholders inspect layer functions and understand model decisions and actions.Transparent operation can support trust, accountability, auditing, and error correction.Source: 14) Interpretability · p. 6

B. Design Options of Guardrails

1) Actions:

  • Actions also include filtering, flagging, modifying, validating, making parallel calls, retrying, and falling back when a workflow step fails.Validation checks artifacts against predefined criteria; fallback redirects to a previous step and state.Source: 1) Actions · p. 6
  • Filtering, flagging, modifying, validating, parallel calls, retry, and fallback provide further ways to scan, mark, adjust, check, repeat, or redirect processing.Parallel calls can send multiple requests and select a better response; retry follows an initial failure or unsatisfactory result.Source: 1) Actions · p. 6
  • Human intervention, deferral, isolation, redundancy, evaluation, and blocking are additional actions for reviewing, postponing, segregating, backing up, assessing, or stopping processing.These actions are described as mechanisms for addressing specific agent-artifact needs.Source: 1) Actions · p. 6

2) Targets:

The section treats guardrail targets as spanning entire pipelines and specific agent artifacts, with Table II mapping targets to actions.

2) Targets:

  • Guardrails may cover an entire pipeline, including prompts, intermediate results, and final results, or focus on particular agent artifacts.Table II is identified as mapping agent targets to guardrail actions; the combinations are illustrative rather than exhaustive or definitional.Source: 2) Targets · pp. 6–7, Table II · p. 7
  • Pipeline-level targets include initial prompts, outputs generated at intermediate workflow stages, and final outputs delivered to users or downstream systems.Monitoring intermediate results can detect anomalies or inaccuracies before they propagate to final results.Source: 2) Targets · p. 7
  • Artifact targets include goals, context, reasoning, plans, workflows, and memory, covering alignment, relevance, reasoning soundness, runtime exceptions, and memory poisoning.Memory guardrails can retain relevant data and discard outdated or irrelevant information.Source: 2) Targets · p. 7
  • Other artifact targets include tools, knowledge bases, other agents, foundation models, and execution time, with guardrails addressing access, validation, interaction risks, safe outputs, and cutoffs.Tool protections may restrict capabilities and detect vulnerabilities; execution cutoffs can halt operations exceeding predefined thresholds.Source: 2) Targets · p. 7

3) Rules:

The section distinguishes rule configurations by uniformity, priority, context dependence, and negotiability.

3) Rules:

  • Uniform rules apply the same guardrails across scenarios and suit stable, well-understood risks, while priority-enabled rules rank guardrails by operational or data criticality and sensitivity.Uniformity can reduce the complexity of managing diverse guardrails.Source: 3) Rules · p. 7
  • Context-dependent rules adjust to operational conditions, while negotiable rules range from flexible soft guardrails to rigid hard guardrails for critical legal, ethical, or safety standards.Soft guardrails balance protection with operational flexibility; hard guardrails are non-negotiable.Source: 3) Rules · p. 7

4) Applicability Scope:

The taxonomy organizes guardrails by where they apply, which data modalities they handle, and the underlying techniques they use. These choices range from regulatory and user-level scope to rule-based, machine-learning, and hybrid designs.

4) Applicability Scope:

  • Guardrail scope spans industry regulation, organizational policy, team constraints, and individual user preferences.These levels cover legal compliance and auditability, internal governance, technical or operational limits, and personalized behavior.Source: 4) Applicability Scope · pp. 7–8

5) Modality:

  • Multimodal guardrails synchronize protections across data types, while single-modal designs address risks specific to one input or output type.Text-based examples include misinformation and privacy; image-based examples include explicit-content detection and image quality.Source: 5) Modality · p. 8

6) Underlying models:

  • Rule-based guardrails use deterministic predefined rules and suit stable operating conditions, but may miss novel scenarios and require updates.Static rules can struggle to detect novel AI-generated content outside their predefined scope.Source: 6) Underlying models · p. 8
  • Machine-learning guardrails adapt to new data and scenarios; narrow models need domain-specific protections, while foundation models require scalable coverage across applications.Foundation-model guardrails can be computationally intensive and require substantial training data.Source: 6) Underlying models · p. 8
  • Hybrid guardrails combine rule-based decisions with machine-learning adaptability to address evolving threats, but can increase system complexity.The cited PaLM 2 example iteratively generates toxic content to test safety guardrails.Source: 6) Underlying models · p. 8

V. REFERENCE ARCHITECTURE FOR DESIGNING MULTI-LAYERED RUNTIME GUARDRAILS OF AGENTS

The reference architecture places runtime guardrails around FM-based agents and their interactions with the external environment. It distributes protections across quality attributes, pipeline stages, and agent artifacts, using overlapping layers to address gaps.

V. REFERENCE ARCHITECTURE FOR DESIGNING MULTI-LAYERED RUNTIME GUARDRAILS OF AGENTS

  • The proposed architecture has four components: the external environment, agent components, built-in multi-layered runtime guardrails, and AgentOps infrastructure.It is derived from the systematic review's findings and recommendations; operational validation is outside the paper's scope.Source: §V · p. 8, Figure 3 · p. 9

A. External Environment:

  • The external environment includes users, other agents, tools, and knowledge bases that interact with the agent at runtime.Users provide goals and contextual inputs; the agent may draw on environmental context and interact with external resources to complete tasks.Source: A. External Environment · p. 8

B. Agent Components

  • Agent components comprise a context engine, reasoning and planning, workflow execution, and memory.The context engine enriches prompts with multimodal context; planning selects steps and resources, execution performs them, and memory stores prior information, plans, and results.Source: B. Agent Components · pp. 8–9

C. Multi-layered Runtime Guardrails

  • Swiss-cheese layers target quality attributes, pipeline stages, and artifacts, with differently positioned gaps allowing another layer to catch issues that pass through one.The architecture operationalizes target mappings from the taxonomy; a layer may, for example, address prompt privacy or tool security.Source: C. Multi-layered Runtime Guardrails · p. 9
  • Guardrails can address accuracy, efficiency, privacy, security, safety, fairness, and compliance at prompts, intermediate workflow results, and final outputs.Prompt checks include sensitive information and harmful content; intermediate checks verify accuracy and safety, while final checks align outputs with goals and governance requirements.Source: C. Multi-layered Runtime Guardrails · p. 9
  • Artifact-level guardrails cover goals, context, memory, reasoning, plans, tools, knowledge bases, other agents, and foundation models.Examples include checking goal feasibility and governance alignment, validating context, screening memory for malicious content, and assessing tool quality.Source: C. Multi-layered Runtime Guardrails · pp. 9–10

D. AgentOps

AgentOps provides observability by monitoring and recording runtime data about agent pipelines, artifacts, and applied guardrails. The recorded evidence includes metadata and can activate relevant guardrails.

D. AgentOps

  • AgentOps continuously records runtime pipeline, artifact, and guardrail data as evidence with metadata such as FM version and timestamp.Collected data can also feed multi-layered guardrails to activate relevant protections as needed.Source: D. AgentOps · p. 10

VI. THREATS TO VALIDITY

The authors describe safeguards used to reduce literature-search and taxonomy-validation bias, while acknowledging limits in coverage, context generalizability, and freshness. They conclude with a 14-quality taxonomy, design options, and a multi-layered architecture, with guardrail services planned as future work.

VI. THREATS TO VALIDITY

  • The literature search targeted guardrails and FM-based agents across commonly used IT and software engineering databases, with manual Google Scholar snowballing and review protocols to reduce selection bias.Source: §VI · p. 10
  • The taxonomy’s definitions and categories may omit relevant aspects, and expert feedback may bias results toward experts’ needs despite review protocols intended to reduce that bias.Source: §VI · p. 10
  • Guardrail generalizability across agent contexts remains limited, specific adaptations may be needed in healthcare or finance, and rapid evolution may make findings outdated.Source: §VI · p. 10

VII. CONCLUSION AND FUTURE WORK

  • The taxonomy identifies 14 qualities and design options covering actions, targets, rules, applicability scope, modality, and underlying models.Source: §VII · p. 10
  • The proposed multi-layered runtime guardrail architecture is based on the taxonomy, SLR findings, and an empirically grounded reference architecture approach.Source: §VII · p. 10
  • Future work plans guardrail services for a scientific agent platform that implement the architecture and integrate the taxonomy’s design options.Source: §VII · p. 10
Loading 2408.02205v4…