PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent libOS: A Runtime Substrate for Capability-Controlled Self-Evolving LLM Agents

Yingqi Zhang

arXiv:2606.03895v2cs.OScs.AIcs.CR

TL;DR

Self-evolving agents can expand their visible actions without a stable boundary separating those actions from resource authority. Agent libOS enforces that boundary with explicit capability checks at runtime, preventing unauthorized effects in deterministic tasks while incurring a 7.0% conservative false-denial rate.

  • Problem

    Model-facing wrappers are interfaces rather than authority boundaries, leaving resource security dependent on consistently correct checks across every tool implementation.

  • Method

    Agent libOS models agents as processes whose self-evolving actions use explicit, auditable capability checks and controlled authority mutation.

  • Results

    On deterministic tasks, Agent libOS produced zero unauthorized effects, with a 7.0% conservative false-denial rate.

  • Takeaways & Limitations

    Self-evolving agents can evolve what they request without silently expanding what they are authorized to affect.

  • Takeaways & Limitations

    The deterministic benchmark isolates runtime boundaries from planner variance and does not establish safety or usefulness in full end-to-end agent evaluations.

Abstract

from arXiv · show

Large language model (LLM) agents are becoming long-running software actors rather than fixed tool users. They accumulate memory, activate skills, synthesize tools, fork children, attach remote resources, and commit checkpoints into reusable execution images. These mechanisms improve adaptability, but also create a systems-security failure mode: if exposing an action also grants the authority needed to perform it, self-evolution becomes a permission-escalation path. This paper presents Agent libOS, an agent-native library-OS substrate for capability-controlled self-evolving agents. Its central invariant is that model-visible affordances may evolve while resource authority changes only through explicit, audited runtime primitives. Agent libOS represents an agent as an AgentProcess with process identity, process-local Object Memory, message queues, a tool table, loaded Skills, process-local Deno/TypeScript JIT tools, child processes, budgets, checkpoints, and explicit capabilities. AgentImage objects define boot-time prompt and tool-table state; Skills and JIT tools extend the action surface; checkpoint-derived images make internal state reusable. None of these mechanisms grants filesystem, shell, human, memory, process, checkpoint, image, JSON-RPC, MCP, or PTY authority by itself. The prototype implements process-local namespaces, persistent runtime state, LLM-call observability, human approval queues, budgets, syscall-mediated JIT tools, trusted Runtime Modules, Object-bound PTY sessions, checkpoint restore/fork/commit, JSON-RPC and MCP providers, and a deterministic runtime-safety benchmark. On 27 versioned deterministic tasks, it completed the task plans while preventing all modeled unauthorized side effects, with a 7.0% conservative false-denial rate. Simple wrapper and sandbox baselines preserved task completion but failed most safety checks.

1 Introduction

Agent libOS treats self-evolving LLM agents as long-running software actors whose changing action surfaces must not silently expand resource authority. It introduces a capability-controlled runtime substrate in which protected effects occur through identity-, capability-, primitive-, policy-, approval-, budget-, and audit-mediated boundaries.

  • Motivation: Self-evolving agents can persist memory, load Skills, generate JIT tools, fork children, request human authority, checkpoint state, and boot checkpoint-derived images.These capabilities make agents resemble long-running software actors rather than single request-response assistants.
  • Problem: Wrapper-level designs can let untrusted text turn Skill activation, JIT registration, or remote-tool visibility into hidden host authority.Confirmation prompts and containers may surround wrappers rather than the resource-touching primitive, so they are not sufficient by themselves.
  • Core invariant: Agent libOS enforces that an agent may change what it can ask for without thereby changing what it is authorized to affect.Protected effects remain inside libOS primitives governed by process identity, typed capabilities, policy, human approval, provider containment, budgets, events, and audit records.
  • Runtime model: The runtime model combines AgentProcess identity, AgentImage boot state, process-local Object Memory, tools, Skills, JIT tools, budgets, queues, checkpoints, typed capabilities, providers, and append-only audit.The implementation includes syscall-mediated Deno/TypeScript tools, trusted Runtime Modules, optional Object-bound PTY sessions, JSON-RPC and MCP providers, persistent runtime state, and checkpoint-derived images.
  • Evaluation and scope: 27 deterministic tasks anchor a runtime-safety harness with wrapper and sandbox baselines, a side-effect oracle, audit checks, and metrics for unauthorized effects, false denial, approvals, resource use, and audit completeness.The paper positions this contribution architecturally rather than as a new reasoning prompt, role protocol, or tool-synthesis optimizer.

2 Problem Setting, Threat Model, and Design Goals

Agent libOS addresses the security problem created when self-evolving agents change prompts, tools, processes, images, checkpoints, and remote or terminal affordances without changing authority implicitly. Its design therefore separates evolving action surfaces from stable, audited runtime authorization while targeting unauthorized protected effects and preserving provenance for audit.

  • Problem setting: Runtime self-evolution changes non-parametric components such as prompts, Skills, JIT tools, processes, images, checkpoints, remote endpoints, and terminal sessions.These operations resemble loading code, changing an address space, creating child processes, opening devices, and checkpointing execution state more than rewriting prompts.
  • Threat model: Model-facing wrappers are interfaces, not authority boundaries, so direct access to host resources makes security depend on every wrapper author implementing checks correctly.Indirect prompt injection and tool-output injection exploit the gap between untrusted text and trusted tool execution.
  • Design goals: The runtime must let visible actions evolve while requiring every protected effect to pass through stable runtime primitives with authorization rules that do not change when new affordances appear.This separates tool-list, Skill, image, checkpoint, and endpoint evolution from authority acquisition.
  • Threat model: The threat model includes untrusted files, documents, tool outputs, Skills, JIT source, remote responses, and messages steering agents toward unauthorized resource, process, checkpoint, image, remote, or approval effects.The trusted computing base is limited to runtime managers, configured provider backends, the host-selected LLM profile registry, and operator-accepted startup Runtime Modules.
  • Design goals: The goal is to prevent unauthorized protected effects and retain provenance for auditing allowed and denied effects, without solving semantic prompt injection, rolling back irreversible external effects, or making untrusted Runtime Modules safe.Checkpoint restore reconstructs scoped runtime state and reports provider-classified external effects.

3 System Model

Agent libOS models each agent as a process with local state, tools, skills, budgets, queues, children, and explicit capabilities. Its core security invariant separates model-visible action surfaces from resource authority, requiring protected effects to cross an audited primitive boundary.

  • Process model: An Agent libOS process comprises an AgentImage, lifecycle state, Object Memory, working directory, tool table, Skills, JIT tools, capabilities, budgets, queues, and child processes.These components define process-local execution state and authority-related records.
  • Authority invariant: Changing visible images, checkpoints, remote endpoints, tools, or Skills does not change resource authority without explicit capability operations or audited approval.Capability changes retain issuer lineage and are not hidden side effects of tool registration or Skill loading.
  • Authority invariant: AgentImages specify boot-time prompts, tools, Skills, JIT tools, required capabilities and modules, profiles, policies, metadata, and workspace seeds, but execution and checkpoint-derived boot do not mint capabilities.Trusted module identity and source-hash requirements can cause boot to fail closed.
  • Primitive boundary: Every primitive normalizes requests, evaluates capabilities and policy, applies deterministic constraints, handles approval, charges budgets, invokes providers, and emits audit records before or after protected effects.The same sequence applies across wrappers, Deno/TypeScript JIT syscalls, CLI workflows, the GUI, and trusted startup modules; wrappers expose actions but do not own authority.

4 Architecture and Authority Model

Agent libOS separates model-visible action surfaces from runtime-owned authority and host effects. Structured capabilities, explicit primitive-mediated mutation, approval controls, and audited providers constrain self-evolution without making tools or backends security bypasses.

  • Runtime structure: The runtime layer owns identity, authority, scheduling, context materialization, checkpoints, wakeups, and audit, while skills and tools provide ergonomic model-selectable actions.Providers perform concrete host calls but must not alter tool schemas or bypass primitive checks.
  • Capability model: Capabilities are typed, durable authority records over canonical resources, with rights and allow, deny, or ask effects.Deny dominates matching allows; unknown rights and constraints fail closed, and wildcards are typed and terminal.
  • Capability mutation: Capability mutation requires explicit trusted, issuer, admin, or grant authority, while delegation may attenuate scope but cannot launder deny or ask boundaries or transfer finite-use rights.Fork and spawn inherit authority only through explicit delegation.
  • Human approval: Human approval is a blocking primitive and audit-recorded state transition, but ordinary approval cannot grant root filesystem write, shell wildcard execution, or capability-admin authority.Approved filesystem writes use scoped one-shot capabilities, while other processes continue running.
  • Mediated execution: Shell, JIT, and remote-provider operations remain mediated by process-local resolution, deterministic policy, syscall protocols, primitive capabilities, and pre-invocation authority checks.JIT tools lack Deno host permissions, and JSON-RPC/MCP callers provide stable endpoint or server identifiers rather than URLs, credentials, or raw wire methods.

5 Self-Evolution Mechanisms

Agent libOS supports self-evolution by changing model-visible affordances and reconstructable internal state through Skills, JIT tools, checkpoints, images, and Object Memory. These mechanisms remain non-authoritative because resource access and side effects require explicit capabilities and runtime-mediated primitives.

  • Skills: Skills extend process prompts and tool visibility while remaining non-authoritative; advisory capability metadata cannot create capabilities, and unloading removes only their contributed context and tools.Activation validates bundled TypeScript JIT tools and binds package contents with hashes.
  • JIT Tools: JIT tools follow propose, validate, and register phases, with validated tools added only to the registering process and runtime access mediated through libos.syscall.Validation includes static checks, import allowlists, size limits, and configured sandbox tests.
  • Images and Checkpoints: Checkpoints and images make reconstructable runtime state reusable through scoped inspection, forking, restoration, and commit without rolling back recorded history or external side effects.Checkpoint state includes process data, memory, namespaces, capabilities, tools, JIT candidates, Skills, mailboxes, and module identifiers.
  • Object Memory: Object Memory is a typed, capability-controlled graph whose namespace-local names are not capabilities, while lookup and materialization require separate authority.Released objects disappear from lookup, listing, and materialization, and release revokes stale object capabilities.
  • Object Memory: Process-local namespaces, attenuated memory views, explicit child-memory merges, and budgeted context materialization constrain how self-evolution exposes and transfers internal state.File/object bridge tools move content without returning full payloads in prompt-visible tool results.

6 Implementation

The current Agent libOS implementation is a Python research prototype with process-local runtime features, audited LLM and tool observability, and a host-facing supervision console. Authority remains mediated by explicit runtime checks rather than being granted by API or GUI visibility.

  • Implementation coverage: The current implementation is a Python research prototype covering the Agent libOS components summarized in Table 2.The supplied implementation section identifies the prototype and refers to Table 2 for its component coverage.
  • Implementation coverage: AgentProcess stores an llm_profile_id, while the host resolves profiles at call time and forked or child processes inherit the parent profile by default.Root spawn selects an explicit host profile, image default, or runtime default; exec preserves the current profile unless overridden by the host.
  • Observability: Every LLM action-selection call is persisted with provider identifiers, API mode, available token usage, errors, and bounded observability envelopes.The envelopes can include prompts, visible schemas, model output, tool calls, reasoning metadata, and raw provider responses; full LLM input/output persistence is enabled by default in controlled deployments.
  • Observability: Tool calls, JIT syscall arguments, validation logs, and failed results are recorded in bounded or redacted envelopes, while full tool results are stored as size-limited Object Memory objects.Sensitive fields are not persisted raw in audit or event records, and larger data should move by file or object reference.
  • Supervision and authority: The Electron console is a host-facing supervision surface, and high-risk operations require explicit confirmation while actor-process requests are checked against that process’s capabilities.API and GUI visibility never grants process authority; requests without an actor process id run as an audited local admin actor.

7 Evaluation

The evaluation tests whether Agent libOS preserves runtime safety, useful completion, auditability, and bounded overhead as agents evolve their action surfaces. Across 27 deterministic tasks, all runners achieved task success, while the key distinction was preventing forbidden effects and preserving provenance.

  • Evaluation questions: The evaluation targets unauthorized side effects, useful completion under policy, audit completeness, and the cost of the runtime boundary.These questions cover visible tools, Skills, JIT tools, images, child processes, checkpoint forks, remote endpoints, approvals, provenance, and operational overhead.
  • Benchmark and runners: The benchmark contains 27 deterministic YAML tasks spanning secret access, forbidden filesystem and shell effects, authority leakage, and self-evolution mechanisms.Each task specifies goals, fixtures, attack classes, allowed and forbidden effects, safety oracles, capabilities, policies, scripted responses, audit conditions, and mock actions.
  • Results: 100% task success was recorded for every runner because all followed the same deterministic task-level action plans, making safety and provenance the differentiators.Task success means reaching the declared high-level goal under the deterministic plan and does not itself imply safety.
  • Results: 72 tool calls, 13 primitive-level effect attempts, 140 synthetic LLM tokens, and 47.5 seconds of wall-clock runtime were recorded for the Agent libOS row.The wall-clock figure is environment-dependent and is reported only as a smoke-cost indicator; the run also exercised self-evolution surfaces.
  • Results: 7.0% false denials arose from three self-evolution attempts requiring additional explicit bootstrap authority despite allowed high-level behavior.The denials were conservative and indicate opportunities to improve package-registration and child-delegation ergonomics without weakening the primitive boundary.
  • Scope and limitations: The evaluation supports only the narrower claim that primitive-level capabilities prevent modeled runtime effects from silently expanding authority, not full end-to-end agent safety.Real-model prompt-injection workloads, adversarial remote providers, Git/worktree providers, long-running context growth, audit-query latency, and production-grade sandbox backends remain unevaluated.

8 Related Work

Agent libOS builds on research in self-evolving agents, agentic workflows, agent benchmarks, and OS support for LLM agents while targeting a distinct runtime-security problem. Its focus is preserving authority invariants as agent capabilities and runtime mechanisms evolve, without claiming to solve prompt injection semantically or provide kernel-level isolation.

  • Self-evolving agents: Self-evolving-agent research covers model, context, tool, and architecture evolution, alongside test-time and reward-, demonstration-, and population-based updates.The paper uses Gao et al.’s taxonomy and cites systems including Voyager, AFlow, and SkillWeaver.
  • Agentic systems: Prior agent systems establish patterns for reasoning, tool use, collaboration, workflow roles, and agent-computer interfaces, while CodeAct enables executable, revisable code actions.The cited systems include ReAct, Toolformer, ToolLLM, HuggingGPT, Reflexion, AutoGen, MetaGPT, CAMEL, AgentScope, SWE-agent, and OpenHands/OpenDevin.
  • Evaluation: Agent libOS complements task benchmarks by measuring whether evolving Skills, JIT tools, images, checkpoints, processes, endpoints, and GUI actions preserve authority invariants.SWE-bench, WebArena, and OSWorld evaluate useful task completion in external environments, whereas Agent libOS evaluates runtime safety during evolution.
  • Agent-runtime systems: Unlike general LLM-agent OS work, Agent libOS operates at the agent-runtime layer with agent-native resources including tool tables, Skills, Object Memory, images, checkpoints, and queues.MemGPT frames context management as virtual memory, while AIOS studies scheduling, context, memory, storage, access control, and LLM/tool resource management.
  • Security and systems foundations: Agent libOS limits the consequences of prompt injection through capabilities, policy, approval, budgets, and primitives, while borrowing from capability systems and library-OS traditions without implementing a kernel or formal verification.The paper distinguishes consequence control from semantic prompt-injection solutions and excludes hardware isolation and POSIX compatibility.

9 Discussion

Agent libOS frames self-evolution as an agent-native operating-system layer, while emphasizing that stochastic execution and imperfect isolation require explicit, auditable authority boundaries rather than relying on the analogy alone.

  • Library-OS analogy: Agent libOS localizes self-evolution above the host OS by managing agent-native resources such as tools, memory, identity, checkpoints, images, skills, JIT code, and remote handles.This layer provides the stable boundary that wrapper-level designs lack.
  • Providers: Providers may vary from local filesystems and containers to JSON-RPC, MCP, browsers, databases, or Git, while preserving identity, capability checks, policy, events, audits, and effect classification.The provider mechanism separates resource implementation from authority enforcement.
  • Limitations: LLM processes are stochastic, prompt-sensitive, model-mediated, semantically validated, and dependent on slower human approval, so the OS analogy guides design but does not prove safety.Their memory is selected, summarized, compacted, and materialized rather than byte-addressed.
  • Implementation limits: Deno no-permission execution is not a formal production sandbox; stronger isolation may require containers, WASM, microVMs, remote sandboxes, or provider-specific mechanisms.Trusted Runtime Modules expand the trusted computing base, and checkpoint restore cannot reverse irreversible external effects.
  • Authority boundaries: Self-evolution can amplify failures by creating powerful tools, persisting unsafe procedures in images, or moving effects through remote endpoints, requiring visibility, scope, reversibility where possible, and auditability otherwise.The design principle is to evolve affordances without implicitly expanding authority.

10 Conclusion

Agent libOS argues that self-evolving agents need a runtime substrate where model-visible affordances can change without creating hidden permission grants. It enforces this boundary through process identity and capability checks at primitive use, rather than prompt rules or wrapper conventions.

  • 10 Conclusion: Agent libOS separates changing model-visible affordances from permission grants through a runtime substrate for self-evolving agents.The conclusion states that smarter planning and larger tool catalogs alone are insufficient.
  • 10 Conclusion: The substrate treats agents as processes with explicit identity, Object Memory, tools, Skills, JIT tools, images, checkpoints, queues, budgets, capabilities, human queues, classified effects, and append-only audit.These process components define the runtime state and control surface described in the conclusion.
  • 10 Conclusion: Its central security boundary is process identity plus capability checks at primitive use, not prompt rules or wrapper conventions.The conclusion presents this boundary as demonstrated by the current artifact on deterministic tasks involving Skills, JIT tools, images, checkpoints, and child processes.

Artifact Availability

The implementation, documentation, benchmark harness, and GUI are publicly available in the Agent libOS GitHub repository. The repository includes commands for running the deterministic demo and benchmark smoke path, with documentation identifying the M1 harness as an early deterministic workload.

  • Artifact Availability: The implementation, documentation, benchmark harness, and GUI are available in the public Agent libOS repository.Repository: https://github.com/yingqi-z20/Agent-libOS
  • Artifact Availability: The deterministic demo runs with the command `uv run agent-libos demo`.
  • Artifact Availability: The benchmark smoke path runs `run_benchmark.py` over `benchmarks/runtime_safety` and collects metrics with `collect_metrics.py`.The full Agent libOS runner is used, and the documentation identifies the M1 harness as an early deterministic workload.
Loading 2606.03895v2…