PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentKernel: The Trust-Native Agentic Operating System

Zhenhua Zou, Sheng Guo, Qiuyang Zhan, Lepeng Zhao, Shuo Li, Zhuotao Liu

arXiv:2609.29647v1cs.CRcs.AI

TL;DR

AI agents combine untrusted inputs, persistent memory, and privileged tools, while existing stacks lack a unified mandatory layer for mediating trust across the lifecycle. AgentKernel proposes an external security kernel with four lifecycle pillars and layered semantic and syscall enforcement. Its architecture specifies independent controls, but LLM-based enforcement components remain an uncharacterized risk and mediation depends on agents using only the adapters.

  • Problem

    Agent workloads need mandatory lifecycle mediation because untrusted inputs, memory, and privileged actions cross semantic trust boundaries that ordinary syscall controls do not capture.

    Source: §1 · p. 2, §2.2 · p. 6

  • Method

    AgentKernel wraps the agent core in a mandatory enforcement boundary with independent Identity, Perception, Cognition, and Execution controls.

    Source: §4.1 · pp. 10–11

  • Results

    The design specifies independent pillar invariants and a policy-intersection rule under which effective permissions can only narrow.

    Source: §3 · p. 8, §5.2 · p. 18

  • Takeaways & Limitations

    The architecture places every inbound interaction, outbound action, and agent-to-agent communication through the security kernel.

    Source: §4.1 · p. 11

    Mediation is incomplete if agents retain access to models, tools, or storage outside the three adapters, and deployment must enforce this assumption.

    Source: §5.5 · p. 21

Abstract

from arXiv · show

Modern AI agents routinely cross trust boundaries: they ingest untrusted content, combine it with privileged instructions, persist intermediate beliefs in long-term memory, and invoke privileged tools. This creates an attack surface in which malicious payloads can enter through model inputs and cause harmful tool actions. Yet current governance stacks remain application-level middleware that share a process trust boundary with the agents they monitor. We argue that agents need an operating-system substrate providing mandatory, non-bypassable services for identity, input mediation, memory governance, and execution control. We introduce AgentKernel, a trust-native agent operating system built around the premise that security must be a first-class design constraint. AgentKernel wraps the agent lifecycle in a mandatory enforcement boundary organized into four pillars: Identity, Perception, Cognition, and Execution. Each pillar adapts classical OS security principles to failures at the semantic plane, including delegation abuse, prompt injection, memory poisoning, and tool misuse. AgentKernel treats structural security as a capability multiplier. Kernel-managed identity supports trustworthy cross-organization collaboration; graduated perception replaces brittle single-point filters; information-flow-controlled memory improves retrieval fidelity while limiting poisoning; and semantic-to-kernel enforcement permits broader tool privileges behind a non-bypassable boundary. We position AgentKernel as the missing OS layer beneath orchestration frameworks, agent runtimes, governance platforms, and execution sandboxes, and use systematic comparison and security analysis to show how a single integrated architecture can enforce security across the full agent lifecycle.

1 Introduction

AI agents combine untrusted context with high-privilege tools, but lack a unified OS layer to mediate trust across the lifecycle. AgentKernel proposes a mandatory semantic security kernel that complements the traditional OS and unifies identity, perception, cognition, and execution governance.

The structural gap

  • Agent stacks couple an LLM to multi-source context and high-privilege tools, while lacking an operating system designed to mediate that fusion end to end.The missing substrate is framed as an infrastructure gap that also caps reliability, auditability, and cross-agent collaboration at scale.Source: §1 · p. 2
  • Existing orchestration, runtime, governance, and sandbox systems address slices of the stack, but are not specified as one mandatory kernel mediating every lifecycle crossing.Governance can share a process trust boundary with agent logic, while optional middleware may not bind actual syscall behavior.Source: §1 · p. 2
  • A DevOps example traces how an injected PR comment can pass through unauthenticated delegation, reach the model, poison memory, and influence a deploy action.The paper frames these as missing transitions in one end-to-end mediation story, rather than unrelated bugs.Source: §1 · pp. 2–3

Proposal and goals

  • AgentKernel is proposed as a mandatory layer for trust moving through delegation, context assembly, durable memory, and tool-mediated side effects.Its practical goals are auditable, composable, scalable sessions under richer inputs, broader tools, and cross-organization collaboration.Source: §1 · p. 3

Design at a glance

  • The four pillars govern the agent lifecycle: kernel-managed identity, graduated input mediation, provenance-aware memory, and execution controls linked to syscall enforcement.The architecture complements rather than replaces the traditional OS kernel, which remains authoritative for low-level isolation and host resources.Source: §1 · pp. 3–4

Contributions

  • The paper contributes a lifecycle diagnosis, a four-pillar architecture and principles, a structured security analysis, and a systematic comparison with representative agent systems.The comparison aims to show where mandatory lifecycle mediation is absent, not to claim feature parity.Source: §1 · pp. 4–5

2 Background and Motivation

Agents repeatedly cross trust boundaries as they ingest information, reason and maintain memory, and act through tools or delegation. The section maps the OS services needed at those crossings and explains why syscall-level controls alone leave semantic gaps with security and capability consequences.

2.1 The Agent Lifecycle

  • The agent lifecycle comprises perception, cognition, and execution, with identity cutting across all three; transitions between input, protected core, and output domains require mediation.Contemporary frameworks often leave those crossings unguarded.Source: §2.1 · p. 5

2.2 What Agents Need from an Operating System, and Why Classical Primitives Fall Short

  • Agents need cryptographic identity, mandatory input mediation, provenance-aware memory, and execution governance, while traditional OS controls operate on syscalls rather than agent meaning.A well-formed tool response can carry prompt injection, and syscall checks may not reveal that a file write came from poisoned memory.Source: §2.2 · pp. 5–6

2.3 Infrastructure Gaps and Their Consequences

  • Missing semantic services pair lifecycle security vulnerabilities with capability limitations, as summarized across the four dimensions in Table 1.The section calls for one mandatory mediation layer spanning identity, perception, cognition, and execution.Source: §2.3 · pp. 6, 8, Table 1 · p. 7
  • Identity gaps permit impersonation and supply-chain injection, while perception gaps leave external content unlabeled and can lead teams to strip multimodal context or add human review.The identity discussion also links presentation-layer spoofing to downstream perception and execution errors.Source: §2.3 · p. 7
  • Persistent poisoned memory can affect later sessions, while broad, slowly changing tool permissions and weak plan–trace alignment threaten execution control and auditability.The section describes these as cognition and execution gaps, respectively.Source: §2.3 · pp. 7–8

3 Design Principles

AgentKernel adapts classical OS security principles to the agent lifecycle, placing an independent mandatory kernel outside the manipulable reasoning loop. The principles combine strict policy composition, independent defenses, semantic-to-syscall enforcement, and a narrow integration boundary.

Principle 1: Security as an Independent & Mandatory Trusted Layer

  • The kernel acts as an external, non-bypassable reference monitor outside the LLM context and controls the adapters through which agents reach models, tools, and storage.This separation prevents a prompt injection from simultaneously subverting security logic embedded in the reasoning context and the protected action.Source: §3 · p. 8

Principle 2: Deny-by-Default with Policy Intersection

  • Effective permissions are the intersection of applicable policies, so a permissive source cannot override a restrictive one.The rule applies across capability chains, skill permissions, and cross-agent session tokens.Source: §3 · p. 8

Principle 3: Defense-in-Depth Across the Agent Lifecycle

  • Identity, perception, cognition, and execution provide independent lifecycle defenses, so a compromise in one pillar does not automatically collapse the others.The pillars target distinct threats and pass provenance labels downstream for informed decisions.Source: §3 · p. 9, Figure 3 · p. 9

Principle 4: Semantic-Level Mediation

  • Semantic mediation evaluates intents, plans, and memory entries, while OS-level enforcement constrains the resulting system calls.The dual-layer design addresses the gap between an action's meaning and its syscall effects.Source: §3 · p. 9

Principle 5: Narrow Agent–Kernel Integration Boundary

  • Three adapters bound agent-visible access to model inference, tool execution, and durable memory, supporting complete mediation and kernel–application decoupling.Non-bypassability depends on agents and orchestration fabric lacking parallel access to those resources outside the adapters.Source: §3 · pp. 9–10

4 The AgentKernel Architecture

AgentKernel places a mandatory security kernel around the agent core so interactions cross one boundary, with four pillars enforcing independent invariants. Its architecture applies kernel-managed identity, graduated input mediation, memory information-flow control, and execution governance across the agent lifecycle.

4.1 Architectural Overview

  • The kernel mediates every interaction around the agent core, integrating four services for identity, input, memory, and execution under a mandatory enforcement boundary.These services address failures at the semantic boundary and compose into a coherent trust argument.Source: §4.1 · pp. 10–11, Figure 4 · p. 11

4.2 Pillar 1: Identity as a Kernel-Managed Resource

  • Kernel-managed identity keeps private keys outside application code and makes identity a live enforcement input across the agent lifecycle.The AIC capability boundary constrains tool permissions and eBPF allowlists; identity signals also inform perception, memory provenance, and signed execution traces.Source: §4.2 · pp. 11–13

4.3 Pillar 2: Perception

  • Perception applies graduated defenses before content enters the LLM context, increasing cost and sophistication across layers rather than relying on one brittle filter.Source tags preserve origin and trust metadata; deterministic rules intercept known injection patterns, while semantic and multi-turn detectors address subtler attacks.Source: §4.3 · pp. 13–14

4.4 Pillar 3: Cognition

  • Cognition governs memory as a labeled, auditable data plane, propagating trust and taint at the individual-entry level instead of degrading an entire session to its worst label.It finds minimally sufficient labels for each candidate memory item, preserving usability and provenance precision; retrieved history is marked as nonexecutable data.Source: §4.4 · pp. 14–15

4.5 Pillar 4: Execution

  • Execution bridges model intent and system effects through deterministic policy checks, optional LLM escalation, kernel-level resource enforcement, and post-execution plan–trace verification.The eBPF hook manager applies allowlists to tool process trees, while workflow permission composition uses the intersection—not the union—of skill permissions.Source: §4.5 · pp. 15–16
  • Execution remains deterministic-first: allowlists and kernel enforcement govern the main path, with LLM adjudication reserved as an optional edge-case enhancement.Concurrent contexts receive separate execution tokens and dedicated BPF namespaces and process trees.Source: §4.5 · p. 16

5 Security Analysis

The section lays out the security analysis, moving from threat assumptions to pillar invariants, composition, architectural choices, and residual risks.

5 Security Analysis

  • The security analysis specifies threat assumptions, argues the soundness of each pillar’s invariants, examines end-to-end composition and distinguishing architectural choices, then accounts for residual risks.The roadmap covers §§5.1–5.5.Source: §5 · p. 16

5.1 Threat Model and Trust Assumptions

AgentKernel’s guarantees are framed against an adversary that can manipulate inputs, compromise tools, poison memory, and attempt impersonation, but cannot bypass mandatory mediation or defeat cryptographic assumptions. The analysis also states trusted-component assumptions and excludes physical, side-channel, and host-kernel attacks, as well as policy-compliant reasoning errors.

5.1 Threat Model and Trust Assumptions

  • The threat model allows attackers to manipulate inputs, compromise tools, poison memory, impersonate agents, inject malicious skills, and exploit operator misconfiguration.The model assumes attackers cannot break standard cryptography or bypass the mandatory mediation boundary.Source: §5.1 · pp. 16–17
  • Guarantees depend on protected private keys, adapter-only access to inference, tools, and durable memory, unmodified eBPF probes, and an uncompromised GAR signing key.The trusted computing base includes GAR, AgentKernel, three adapters, the OS kernel’s eBPF subsystem, and cryptographic primitives.Source: §5.1 · p. 17
  • Physical and side-channel attacks, host OS kernel compromise, and policy-compliant but poor LLM decisions are outside the security threat model.The paper classifies reasoning errors that violate no stated invariant as reliability issues rather than security violations.Source: §5.1 · p. 17

5.2 Security Invariants

AgentKernel specifies security invariants for each lifecycle pillar, with mechanisms and soundness arguments intended to make mediation enforceable across identity, perception, cognition, and execution.

Identity invariants

  • Identity relies on GAR-only AIC issuance to prevent agent self-certification, while kernel enforcement keeps delegated capability boundaries from widening.Kernel-mediated signing and single-use nonces provide non-repudiation and freshness, respectively.Source: §5.2 · p. 17

Perception invariants

  • Perception requires every item reaching the LLM context to carry provenance, while orthogonal pipeline layers provide independent barriers against evasion.The source tagger is the mandatory first stage; syntactic-filter evasion still faces semantic and temporal layers.Source: §5.2 · p. 17

Cognition invariants

  • Cognition assigns memory-item taint labels that dominate contributing source labels, but this guarantee depends on accurate source segmentation.Adversarial content that blurs segment boundaries can degrade label fidelity; explicit, auditable cross-agent sharing addresses silent contamination propagation.Source: §5.2 · pp. 17–18

Security invariants

  • Table 2 consolidates the architecture’s security invariants.The paper states that each pillar’s invariants are paired with an enforcement mechanism and a brief soundness argument.Source: §5.2 · p. 17, Table 2 · p. 18

Execution invariants

  • Execution composes policies by intersection, installs dynamically computed allowlists as eBPF probes for the process tree, and surfaces plan–trace mismatches for audit.The mismatch check detects rather than prevents divergence.Source: §5.2 · p. 18

5.3 Composition and Cross-Pillar Guarantees

The four pillars compose through shared provenance metadata, while each revalidates upstream claims to limit end-to-end compromise; their distinct mechanisms reflect different threat types.

Composition and cross-pillar guarantees

  • Identity establishes content provenance, Perception assigns initial trust, Cognition propagates taint through memory derivations, and Execution uses those labels to constrain tool actions.The defined provenance chain creates both composition benefits and coupling risks.Source: §5.3 · p. 18

Layered defense

  • An end-to-end attack must defeat checks across identity, perception, cognition, and execution, with later controls providing backstops when earlier ones miss.For example, execution remains constrained by the agent’s capability boundary and the tool manifest even when cognition propagates contaminated labels.Source: §5.3 · p. 19

Coupling and independence

  • The pillars share provenance, so upstream failures can affect downstream decisions, but each pillar independently revalidates claims against its own invariants.Perception cross-checks identity, cognition checks provenance against source segments, and execution checks taint labels against declared manifests.Source: §5.3 · p. 19

Why four pillars, not one

  • The four-pillar decomposition reflects heterogeneous security problems: cryptography for identity, classification for perception, information flow for cognition, and mediation for execution.The paper argues that a single layer would force a least-common-denominator approach that weakens guarantees at boundaries.Source: §5.3 · p. 19

5.4 Architectural Distinctions

AgentKernel’s architectural distinctions target three different gaps: policy misconfiguration, coarse memory taint, and the separation between semantic and syscall enforcement.

5.4.1 Structural vs. Policy-Based Least-Privilege

  • Capability sets only narrow along delegation chains, so no policy change at one stage can exceed the preceding stage’s signed boundary.This structural attenuation prevents any single stakeholder from unilaterally escalating privileges and supports cross-organizational delegation.Source: §5.4.1 · p. 20

5.4.2 Item-Level vs. Session-Level Taint

  • Item-level taint avoids session-level over-taint by retaining minimal labels per memory item, but adversarially inaccurate source segmentation can understate taint.Session-level taint is sound but can make information-flow control impractical; AgentKernel accepts a bounded soundness trade-off for usability, with execution constraints as an independent backstop.Source: §5.4.2 · p. 20

5.4.3 The Semantic–Syscall Bridge

  • Semantic checks can miss malicious tool behavior, while syscall checks cannot distinguish legitimate actions from actions driven by poisoned memory.The two-phase protocol evaluates intent and provenance to compute an allowlist, then installs it as eBPF probes before the tool process tree starts.Source: §5.4.3 · pp. 20–21
  • The resulting syscall closure constrains a tool and its descendants to the computed allowlist, combining intent-awareness with non-bypassability.The paper distinguishes this combination from sandboxes, which provide non-bypassability without intent-awareness, and guardrails, which provide intent-awareness without non-bypassability.Source: §5.4.3 · p. 21

5.5 Residual Risks and Limitations

AgentKernel’s stated guarantees have residual risks and deployment boundaries, which the authors say bound confidence in how those guarantees should be interpreted.

Taint fidelity

  • Adversarial content that blurs trusted and untrusted source boundaries can make item-level taint labels under-approximate true taint.Robust segmentation remains an open research problem.Source: §5.5 · p. 21

Semantic firewall accuracy

  • The LLM-based semantic firewall trades false positives against false negatives, and adversarial payloads may exploit its classifier blind spots.The graduated pipeline reduces this risk through defense-in-depth but does not eliminate it.Source: §5.5 · p. 21

GAR as a single point of trust

  • Compromise of GAR’s signing key would enable arbitrary AIC forgery and collapse the identity pillar.Cross-GAR federation could distribute this trust, but introduces cross-certification challenges.Source: §5.5 · p. 21

Non-bypass assumption

  • Mediation is incomplete if agents retain ambient credentials or access to filesystems or tool transports outside the three adapters.Enforcing the assumption that adapters are the only access paths is a deployment responsibility.Source: §5.5 · p. 21

Portability and formal verification

  • Execution’s eBPF dependency limits native support on macOS and Windows, where alternative backends must provide equivalent non-bypassability.The stated invariants have not been mechanically verified; the authors defer formal proofs to companion publications.Source: §5.5 · p. 21

Scope of limitations

  • The authors characterize these limitations as boundaries on confidence in the guarantees, with each corresponding to an active research direction.The comparison table’s caption describes AgentKernel’s support across systematic Agent OS dimensions, but does not resolve these residual risks.Source: §5.5 · p. 21, Table 3 · p. 22

6 Related Work: From Agent Harness to Agent OS

AgentKernel argues that the agent harness ecosystem needs an integrated OS layer with mandatory identity, perception, cognition, and execution services. Its comparison distinguishes that lifecycle-spanning mediation from the capabilities and enforcement boundaries of existing infrastructure.

6 Related Work: From Agent Harness to Agent OS

  • The section compares six representative systems across 15 architectural dimensions and qualitatively rates support as comprehensive, partial, or absent.The ratings reflect the authors’ reading of public documentation and repositories, not independent audits.Source: §6 · pp. 22, 24

6.1.1 Orchestration Frameworks

  • Orchestration frameworks compose workflows and route tools, while AgentKernel supplies a complementary OS layer that those frameworks do not provide.Three adapter interfaces let frameworks integrate without replacing their orchestration role.Source: §6.1.1 · p. 24

6.1.2 Agent Runtimes

  • Agent runtimes improve capabilities such as scheduling, memory, integrations, and isolation, but their security enforcement varies in scope and bypass resistance.AgentKernel’s LLM, tool, and storage adapters are the only paths to the guarded capabilities, preserving non-bypassability across deployment forms.Source: §6.1.2 · p. 26

6.1.3 Governance Platforms

  • Governance platforms offer policy and observability features, but co-located governance hooks can be bypassed by code in the same process.The authors position application-level policy systems as complementary: their policy language can run above AgentKernel’s separate enforcement kernel.Source: §6.1.3 · pp. 26–27

6.1.4 Execution Sandboxes

  • Execution sandboxes provide process isolation, whereas AgentKernel adds graduated, context-aware enforcement based on identity, input provenance, and memory taint.Its execution pillar subsumes sandbox functionality and extends it with semantic context.Source: §6.1.4 · pp. 27–28

6.2 The Missing OS Layer: Why an OS, not just a stronger harness?

The surveyed tiers lack a unified OS layer integrating identity, perception, cognition, and execution under shared provenance and taint metadata. AgentKernel is proposed as that mandatory mediation layer, complementing existing agent infrastructure.

6.2 The Missing OS Layer: Why an OS, not just a stronger harness?

  • Across the four harness tiers, no tier supplies unified OS services spanning all four pillars with shared provenance, trust signals, and taint metadata.The authors characterize this as a missing architectural layer rather than a missing individual feature.Source: §6.2 · p. 28

6.3 AgentKernel as the Agent OS Kernel

  • AgentKernel’s kernel comprises a remote Global Agent Registry, a local Agent Kernel, and three adapters that mediate capability interactions.Identity infrastructure, lifecycle information-flow control, and kernel-enforced execution governance are its core services; the framework can be swapped without reimplementing them.Source: §6.3 · p. 28

6.4 Implications for A2A and H2A Interactions

  • For cross-organization agent collaboration, mutual attestation establishes identities and session tokens scope interactions; for human-to-agent use, plan–trace alignment, immutable logs, and operator identity support accountability.Source: §6.4 · p. 28

6.5 Other Related Areas

  • AgentKernel is positioned beneath GUI agents and multi-agent platforms, adds memory-security enforcement to memory systems, and complements protocols that specify what agents can say.The paper contrasts protocol-defined communication with AgentKernel’s concern for what agents can do.Source: §6.5 · p. 29

7 Discussion and Future Directions

The discussion frames AgentKernel as a semantic-plane reference monitor whose non-bypassable security can expand safely delegated capabilities. It also identifies deployment limitations and research needed to establish and operationalize its guarantees.

7.1 Architectural Insights

  • AgentKernel treats the agent as untrusted compute, with provenance connecting identity, perception, memory taint, and execution under a semantic-plane reference monitor.The four-pillar decomposition fits interactive, tool-using agents with persistent memory, but some agent classes may need fewer pillars; cognition’s information-flow guarantees depend on durable memory.Source: §7.1 · pp. 29–30

7.2 Security as a Capability Multiplier

  • The paper argues that stronger non-bypassable security invariants allow broader safe delegation rather than simply constraining agent capability.Examples include cross-organization collaboration, richer inputs, trustworthy long-horizon memory, and expanded tool access; the authors tie these capabilities to the four pillars’ enforcement mechanisms.Source: §7.2 · pp. 30–31

7.3 Integration with the Agent Ecosystem

  • AgentKernel is designed as a minimal, complementary layer beneath existing infrastructure, with adapters defining what it mediates and leaving other concerns to other layers.The paper describes this layered architecture as an explicit commitment, not only an accommodation for existing systems.Source: §7.3 · p. 31

7.4 Limitations and Open Problems

  • Deployability remains constrained by uncharacterized LLM-classifier robustness, a large heterogeneous TCB, unresolved policy-authoring needs, adapter-only-path assumptions in production, and pending realistic performance and security evaluation.The authors specifically call out classifier-failure risk, policy conflicts, ambient credentials or side channels, and measurement of concurrent enforcement overhead and large-store retrieval.Source: §7.4 · p. 32

7.5 Research Agenda

  • The proposed research agenda includes mechanized verification of guarantees and adversarial evaluation of individual pillars and their composition.Targets include formal models of delegation and policy composition, benchmarked perception errors, taint fidelity, plan–trace alignment, and eBPF allowlist races.Source: §7.5 · pp. 32–33
  • Further work includes deployment trust infrastructure, cross-registry and multi-agent security composition, and shared ecosystem standards for identity, provenance, and execution governance.The paper also identifies privacy-preserving attestation, postquantum signatures, and TEE-backed key custody as possible extensions.Source: §7.5 · p. 33

8 Conclusion

Agents need a unified, mandatory layer that mediates identity, perception, memory, and execution across trust boundaries. AgentKernel provides this OS layer beneath existing agent harness components, making security structural while supporting richer capabilities.

8 Conclusion

  • Current orchestration frameworks, runtimes, governance platforms, and sandboxes cover parts of the problem, but none provides unified, mandatory end-to-end mediation.The paper argues agents need an operating system, not only a stronger harness.Source: §8 · p. 33
  • AgentKernel’s Identity, Perception, Cognition, and Execution pillars mediate agent boundaries, applying classical OS security ideas to semantic threats.The cited threats include prompt injection, memory poisoning, and unsafe tool use.Source: §8 · p. 34
  • Structural security is presented as a capability multiplier: kernel-managed identity supports collaboration, graduated perception supports richer inputs, taint-aware memory separates reliable from contaminated knowledge, and kernel-backed execution permits wider tool access.These capabilities retain non-bypassable guarantees.Source: §8 · p. 34
  • AgentKernel complements rather than replaces orchestration frameworks, runtimes, governance platforms, or execution sandboxes, and is presented as a step toward indispensable semantic-level kernel services for higher-stakes tasks.Those services include strong identity, mediated I/O, governed memory, and controlled execution.Source: §8 · p. 34

Related papers

Loading 2609.29647v1…