AIOS: LLM Agent Operating System

• Without effective scheduling, agents can monopolize LLM resources, while GPU-memory retries under concurrent loads degrade throughput.Source: §1 · pp. 1–2

New Agent-serving Architecture

• AIOS separates agent applications from LLM and tool resources into distinct layers to support resource management, efficiency optimization, and safety enhancement.Source: §1 · p. 2

AIOS Kernel Design and Implementation

• The AIOS kernel decomposes agent queries into system calls, schedules their execution, and assigns specialized modules to handle resources and agent operations.Source: §1 · p. 2

AIOS SDK Development

• The AIOS SDK abstracts kernel functions so developers can focus on application logic rather than kernel details.Source: §1 · p. 2

Empirical Results

• AIOS maintains agent performance on standard benchmarks, can improve results on tool-calling benchmarks, and achieves up to 2.1× faster execution across frameworks.Source: §1 · p. 2

• Applications use SDK interfaces and system calls, while the kernel layer separates traditional OS tasks from AIOS services for agent requests.Source: §2 · pp. 2–3
• The application layer provides agent-facing interfaces, the kernel layer manages core functions and resources, and the hardware layer contains physical computing resources.Source: Figure 2 · p. 3
• The AIOS kernel provides LLM cores, scheduling, context switching, memory and storage management, tool handling, and access controls; the hardware layer is accessed through OS system calls.Source: §2 · p. 3

3.1 Relationship and Connection between Modules

• Agent queries are decomposed into thread-bound system calls and routed to module queues; access-related calls are not dispatched by the scheduler.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2 LLM Core

• Each LLM instance is treated as a processing core behind a wrapper with unified inference system calls, accommodating different deployment options.Source: §3.2 · p. 4

3.3 Scheduler

• The scheduler centralizes queues and supports FIFO and Round Robin scheduling, with Round Robin relying on context interruption for LLM inference.Source: §3.3 · p. 4

3.4 Context Manager

• The context manager snapshots and restores generation state to interrupt and resume inference, using decoded outputs for closed-source models or intermediate search trees when logits are available.Source: §3.4 · pp. 4–5, Figure 4 · p. 5

3.5 Memory Manager / 3.6 Storage Manager

• When agent memory exceeds its block limit, LRU-K eviction moves less frequently accessed items from RAM to disk through the storage manager.Source: §3.5 · p. 5, §3.6 · p. 5

3.7 Tool Manager / 3.8 Access Manager

• The tool manager standardizes tool loading and checks parameter and parallel-use constraints, while the access manager enforces privilege groups and user confirmation for potentially destructive operations.Source: §3.7 · pp. 5–6, §3.8 · p. 6

3.9 AIOS SDK

• The SDK exposes kernel functions through APIs and adapts agents from supported frameworks so they can run in AIOS without modifying agent code.Source: §3.9 · pp. 6–7

4.1 Setup

• Experiments use five agent frameworks, three LLM choices, a single A5000 GPU, and a resource-constrained concurrent workload; round-robin is the default scheduler.The workload has one LLM processing one prompt at a time and allows up to 250 concurrent agents by default.Source: §4.1 · p. 7

4.3 Efficiency Analysis (RQ2)

• 2.1× higher throughput for Reflexion agents on Llama-3.1-8b accompanies substantially reduced average agent waiting time across evaluated frameworks.The authors attribute the throughput gain to kernel scheduling that avoids trial-and-error attempts for prompts that cannot be loaded onto the GPU.Source: Figure 6 · p. 8, Figure 7 · p. 8, §4.3 · p. 8

4.2 Agent Performance (RQ1)

• AIOS maintains performance across standard benchmarks and improves it in some cases, with prompt enhancement and tool-call safeguards cited as mechanisms.The evaluation uses HumanEval, MINT code, GAIA, and SWE-Bench-Lite, with success rate as the metric.Source: §4.2 · pp. 7–8

4.4 Scalability Analysis (RQ3)

• As active agents increase from 250 to 2,000, overall execution time and average waiting time remain approximately linear, while the gap between AIOS and non-AIOS widens.The scalability test duplicates HumanEval’s 164 samples and uses Llama-3.1-8b and Mistral-7b.Source: §4.4 · pp. 8–9, Figure 8 · p. 9

Operating systems

• Operating systems developed from batch processing toward time-sharing, multitasking, and modular services such as scheduling, memory management, and filesystem operations.The passage describes LLMs as migrating from application to system layers to provide standardized services across applications.Source: §5 · p. 9

LLM-based agents

• LLM-based agents are studied for planning and reasoning across digital and physical environments, including API use, browsing, code execution, object manipulation, and laboratory work.Source: §5 · p. 9
• Multi-agent systems use cooperative, competitive, or mixed interactions; cooperative examples include role playing, social simulation, and software development.Source: §5 · p. 9

6 Conclusion and Future Work

• AIOS serves LLM-based agents through a kernel that isolates resources and LLM services from applications, complemented by an SDK for accessing kernel functions.Source: §6 · p. 9
• The authors report maintained or enhanced benchmark performance, faster execution, improved throughput, and scalability as concurrent agent loads increase.Source: §6 · p. 9
• The authors envision refining and expanding the architecture to address evolving requirements for developing and deploying LLM-based agents.Source: §6 · p. 10

APPENDIX

• The appendix is organized into kernel implementation, SDK, agent benchmarks, additional experimental results, and discussion sections.Source: APPENDIX · p. 15

A.1 AIOS System Call

• AIOS modules invoke system calls, and each call runs on its own thread to enable concurrent processing.The system-call interface exposes module-specific calls and their arguments.Source: §A.1 · p. 15, Table 2 · p. 15

A.2 LLM Core

• AIOS uses LLMAdapter to provide a unified interface for integrating LLM instances from different backends.Table 3 lists adapted backends and supported features.Source: §A.2 · p. 15, Table 3 · p. 16

A.3 Scheduler

• AIOS builds an extensible scheduler on BaseScheduler, from which FIFO, Round Robin, and priority-based strategies can be implemented independently.The scheduler processes LLM, memory, storage, and tool requests through separate request-processing threads.Source: §A.3 · p. 17

A.4 Context Manager

• The SimpleContextManager enforces time limits while saving and restoring generation state, enabling interrupted LLM tasks to resume without redundant computation.It supports streaming responses and multiple backends, including OpenAI and HuggingFace, and preserves state after at least one token is decoded.Source: §A.4 · p. 17

A.5 Memory Manager

• The memory manager provides session-scoped RAM operations, including CRUD and retrieval, with atomic operations, metadata synchronization, and thread-safe access.Session data is transient and clears when agent sessions end.Source: §A.5 · p. 19

A.6 Storage Manager

• The Storage Manager combines persistent file operations with vector-database semantic search, version history and rollback, thread-safe access, and file sharing.It also supports mounting directories and building a vector database for an agent.Source: §A.6 · pp. 19–20

A.7 Tool Manager

• The tool manager loads and executes tools while preventing tool conflicts.The passage identifies conflict prevention as part of the tool manager’s implementation.Source: §A.7 · p. 21

A.8 Access Manager

• The access manager checks agents’ access to other agents’ resources and requests user permission before irreversible actions such as file deletion.Source: §A.8 · p. 21

A.9 Module Hooks

• A hook mechanism initializes kernel modules and exports their call interfaces, separating module interfaces from implementation details.Source: §A.9 · p. 21

B AIOS SDK

• SDK modules route queries through a central send_request() function to the kernel over HTTP, separating agent logic from requests for LLM, memory, storage, and tool resources.Requests can target localhost or a remote URL.Source: B AIOS SDK · p. 22

B.1 Query and Response

• Query and Response provide the SDK’s foundational structures for exchanging requests and standardized outputs between agent applications and the AIOS kernel.Queries branch into LLM, memory, storage, and tool types, with corresponding response types.Source: §B.1 · pp. 22–23
• The four query types cover LLM interactions, memory operations, persistent storage, and structured calls to external tools.LLM queries support chat, JSON output, tool calls, and file operations; memory queries include adding, retrieving, updating, and removing memories.Source: §B.1 · p. 22
• Matching response types return operation-specific results, completion or status information, and error details.LLM responses can include generated text and tool-call outcomes, while memory responses include content, metadata, and search results.Source: §B.1 · pp. 22–23

B.2 AIOS SDK APIs

• SDK APIs are built on the Query and Response structures and use request() functions to communicate with the kernel.Table 4 lists the available APIs.Source: §B.2 · p. 25, Table 4 · p. 25

B.3 Supported Tools.

• The SDK offers 17 native tools from technology providers, API hubs, and AI model providers, with text and cross-modal capabilities.Twelve tools support text input or output; examples of cross-modal tools include VisualQuestionAnswering, TextToAudio, and VoiceActivityRecognition.Source: §B.3 · p. 25

B.4 Agent Examples

• AIOS SDK examples cover travel planning, recommendations, math, content creation, and academic research tasks.The travel agent coordinates searches for hotels, flights, restaurants, and destination information before assembling a trip plan.Source: §B.4 · pp. 25–26
• The examples pair task-specific workflows with tools such as TripAdvisor, Wikipedia, WolframAlpha, Currency Converter, and Arxiv API.The academic agent gathers tool results to produce an outline or summary; the math agent uses pre-calculation before further operations.Source: §B.4 · p. 27

B.5 Support of Agent Frameworks

• To adapt existing frameworks, AIOS replaces core functions that interact with system resources with native adapter functions.The paper illustrates adaptations for Open-Interpreter, ReAct, Reflexion, MetaGPT, and AutoGen.Source: §B.5 · pp. 28–29
• The demonstrated AutoGen adapter supports AutoGen-0.2 and replaces client and agent methods with AIOS implementations.The text identifies AutoGen-0.2 as the only supported version during the team’s ongoing refactoring work.Source: §B.5 · p. 28

C.1 HumanEval

• HumanEval comprises 164 handwritten programming problems evaluated for functional correctness using test suites averaging 7.7 cases per problem.The benchmark targets natural-language comprehension, logical reasoning, algorithmic thinking, and mathematical operations.Source: §C.1 · p. 30

C.2 MINT

• MINT evaluates multi-turn code generation, decision-making, and reasoning with tool use and natural-language feedback; these experiments use k = 5 and its code-generation subset.MINT contains 586 selected examples, and success rate at interaction limit k is denoted SRk.Source: §C.2 · p. 30

C.3 GAIA

• GAIA contains 466 questions across three difficulty levels, testing everyday reasoning, multimodal processing, web navigation, and tool use.Its scope excludes complex web interactions such as file uploads and posting comments.Source: §C.3 · p. 30

C.4 SWEBench-Lite

• SWE-bench derives 2,294 software-engineering tasks from pull requests in 12 Python repositories, with success determined by test coverage.Figure 10 is captioned as an efficiency analysis across agent frameworks using Llama-3.1-8b on MINT; Table 6 describes scheduling-strategy metrics in seconds.Source: §C.4 · p. 30, Figure 10 · p. 31, Table 6 · p. 31

D.1 Efficiency analysis.

• Efficiency figures cover different agent frameworks on MINT, GAIA, and SWE-Bench-Lite with Mistral-7b or Llama-3.1-8b.The reported comparison includes throughput and latency; the supplied passages identify the figure scope but not specific outcomes.Source: §D.1 · p. 31, Figure 11 · p. 32, Figure 12 · p. 32
• FIFO has the shortest overall execution time, while RR ranks second for execution time and average waiting time but performs better on p90 waiting time.The authors attribute RR’s p90 result to fairer scheduling, which reduces the likelihood of later tasks waiting longer, a pattern that can occur with FIFO.Source: §D.1 · p. 31

D.2 Correctness of context switch.

• BLEU and BERT scores are both 1.0 for final outputs generated with context switching enabled versus disabled.Table 7 describes text-based and logits-based similarity checks between those outputs.Source: §D.2 · p. 31, Table 7 · p. 32

E.1 Ethical Consideration

• The discussion identifies potential benefits in efficiency and productivity, user experience, and an innovation ecosystem for agent developers and researchers.It associates these benefits with automation and resource allocation, context, memory, and storage management, and opportunities for innovation.Source: §E.1 · p. 32
• Potential societal risks include privacy concerns, security threats to sensitive data and operations, and disruptions if integrated systems fail.Source: §E.1 · p. 32
• The authors recommend rules and standards, robust design and maintenance, testing and monitoring, and public engagement to balance potential impacts.The recommendations include backup and recovery plans and developer training.Source: §E.1 · pp. 32–33

E.2 Future Directions

• Future scheduling research could analyze dependencies among agent requests and incorporate locally deployed tool status and snapshots.The proposed direction is a unified scheduling framework for agents and their tools.Source: §E.2 · p. 33
• Context-management work could speed snapshotting and restoration and use compression before snapshotting to reduce storage needs.Source: §E.2 · p. 33
• Future memory and storage designs could share resources among agents and use hierarchical storage to improve retrieval and storage efficiency.The shared approach would let agents benefit from others’ memory or storage; the hierarchical approach would prioritize quicker access and less allocation for frequently accessed information.Source: §E.2 · p. 33
• Safety and privacy directions include protections against attacks and unauthorized access to other agents’ memory, encryption for data transmission, and watermarking developer outputs.The passage presents watermarking as a way to protect intellectual property and trace data lineage.Source: §E.2 · p. 34