MemGPT: Towards LLMs as Operating Systems

1. Introduction

• Fixed context windows limit extended conversations and long-document reasoning; widely used open-source LLMs support only a few dozen exchanges or short documents (Touvron et al., 2023).Source: §1 · p. 1
• Direct context extension incurs quadratic computational and memory costs because of self-attention (Dai et al., 2019; Kitaev et al., 2020; Beltagy et al., 2020).Long-context models can also struggle to use additional context effectively (Liu et al., 2023a), motivating alternatives to context scaling.Source: §1 · p. 1
• Function calls let agents read and write external data, edit their context iteratively, and control transitions between context management, response generation, and user interactions.An agent can repeatedly modify its context while working on a single task and choose when to return a response.Source: §1 · p. 1
• MemGPT emulates virtual memory by letting an LLM retrieve relevant historical data into context and evict less relevant data to external storage.Its memory hierarchy treats the fixed context window as a constrained resource, providing the illusion of longer context without enlarging that window.Source: §1 · pp. 1–2
• MemGPT outperforms existing LLM-based approaches in document analysis and conversational agents by overcoming finite-context limitations.The evaluations target documents exceeding model input capacity and extended conversations requiring context awareness, persona consistency, and long-term memory.Source: §1 · p. 2

2. MemGPT (MemoryGPT)

• MemGPT separates main context, directly accessible during inference, from external context, which must be explicitly retrieved through autonomous memory-management function calls.Main context consists of prompt tokens; external context holds information outside the LLM’s fixed context window.Source: §2 · p. 2

2.1. Main context (prompt tokens)

• Main context combines read-only system instructions, a fixed-size writable working context, and a FIFO message queue containing a recursive summary of evicted messages.Working context is editable only through function calls and can retain user facts, preferences, and agent-persona information. The queue holds conversations, system messages, and function inputs and outputs.Source: §2.1 · p. 2

2.2. Queue Manager

• The queue manager appends incoming messages, triggers inference, saves inputs and outputs to recall storage, and reinserts retrieved messages into the FIFO queue.Retrieved messages are appended to the back of the queue, making them accessible within the LLM’s context window.Source: §2.2 · pp. 2–3
• The queue manager warns of impending context overflow, then evicts messages and updates their recursive summary when the flush threshold is exceeded.Warnings allow the LLM to preserve important information in working context or archival storage before eviction. Evicted messages remain indefinitely in recall storage and can be retrieved through function calls.Source: §2.2 · p. 3

2.3. Function executor (handling of completion tokens)

• The function executor validates LLM-generated calls, executes valid requests, and returns results or runtime errors, while token warnings and paginated retrieval help respect context limits.System instructions describe the memory hierarchy and provide function schemas with natural-language descriptions. Execution feedback lets the processor adjust its subsequent behavior.Source: §2.3 · p. 3

2.4. Control flow and function chaining

• Function chaining immediately resumes inference after flagged calls; yielding instead waits for the next external event.Sequential calls support traversing paginated results and combining information from separate document queries before returning control to the user. User messages, system messages, user interactions, and scheduled events are parsed into plain-text messages for main context.Source: §2.4 · p. 4

3. Experiments

• Experiments evaluate conversational agents and document analysis using GPT-4, GPT-4 Turbo, and GPT-3.5, with augmented datasets released for future research.Conversational tasks extend Multi-Session Chat (Xu et al., 2021), while document benchmarks build on Liu et al. (2023a). Releases include augmented MSC, nested key-value retrieval data, and embeddings for 20M Wikipedia articles.Source: §3 · p. 4

3.1.1. DEEP MEMORY RETRIEVAL TASK (CONSISTENCY).

• MemGPT improves accuracy and ROUGE scores over corresponding fixed-context LLM baselines on questions requiring facts from earlier conversations.Baselines receive lossy summaries of five prior conversations, whereas MemGPT searches the full history through paginated queries. Evaluation uses ROUGE-L recall to accommodate verbose replies and an LLM judge to assess consistency with gold answers.Source: Table 2 · p. 5, §3.1.1 · p. 5

3.1. MemGPT for conversational agents

• Conversational evaluation tests consistency with prior facts and engagement through personalized references to past interactions, using Multi-Session Chat (Xu et al., 2021).MSC contains five sessions of roughly a dozen messages each; the consistency evaluation adds a sixth session containing one question-answer pair.Source: §3.1 · pp. 4–5

3.1.2. CONVERSATION OPENER TASK (ENGAGEMENT).

• MemGPT generates conversation openers that match or occasionally exceed human-written openers, with working-context information storage key to engagement.Openers are compared with gold persona information and human-generated openers using similarity scores. MemGPT’s openers tend to be more verbose and cover more persona information than human openers.Source: Table 3 · p. 5, §3.1.2 · pp. 5–6

3.2.1. MULTI-DOCUMENT QUESTION-ANSWERING.

• MemGPT maintains document-QA performance as available context grows, while document truncation degrades fixed-context accuracy; MemGPT with GPT-4 and GPT-4 Turbo achieves equivalent results.Repeated archival searches and pagination overcome the context-window document limit, but noisy embedding rankings and premature stopping still constrain retrieval. MemGPT performs best with GPT-4; limited function-calling capabilities substantially degrade its GPT-3.5 performance.Source: Figure 5 · p. 6, §3.2.1 · p. 7

3.2. MemGPT for document analysis

• Document analysis motivates flexible memory because documents can exceed context limits and require cross-document reasoning, while larger windows exhibit uneven recall (Liu et al., 2023a).The cited research finds better recall near the beginning and end of a context window than in its middle.Source: §3.2 · p. 6

3.2.2. NESTED KEY-VALUE RETRIEVAL (KV).

• Beyond 2 nesting levels, only MemGPT consistently completes the new nested key-value task; MemGPT with GPT-4 remains unaffected as nesting increases.Repeated function queries access main-context key-value pairs for multi-hop retrieval; evaluation uses 140 UUID pairs and 0–4 nesting levels. GPT-3.5 reaches 0 percent accuracy at 1 nesting level, and GPT-4 and GPT-4 Turbo by 3; MemGPT with GPT-3.5 or GPT-4 Turbo begins declining at 2.Source: Figure 7 · p. 7, §3.2.2 · pp. 7–8

3.2.1. MULTI-DOCUMENT QUESTION-ANSWERING.

• 50 NaturalQuestions-Open questions evaluate document QA with a shared embedding retriever and an LLM judge checking document-grounded answers.Accuracy is evaluated as the number of retrieved Wikipedia documents increases. MemGPT searches archival storage backed by PostgreSQL, pgvector, and an HNSW index, whereas baselines receive independently retrieved top-K documents.Source: §3.2.1 · pp. 6–7

Long-context LLMs

• MemGPT complements longer-context LLMs with hierarchical tiered memory, using the LLM’s context window as main memory.Prior approaches improve context length through sparse attention (Child et al., 2019; Beltagy et al., 2020), low-rank approximations (Wang et al., 2020), and neural memory (Lee et al., 2019). Extending context beyond training lengths (Press et al., 2021; Chen et al., 2023) likewise enlarges MemGPT’s main memory.Source: §4 · p. 8

Retrieval-Augmented Models

• MemGPT’s external memory builds on retrieval augmentation, including methods that actively choose retrievals or interleave retrieval with reasoning.FLARE lets an LLM decide when and what to retrieve during generation (Jiang et al., 2023). Trivedi et al. (2022) interleave retrieval with Chain-of-Thoughts reasoning to improve multi-step question answering.Source: §4 · p. 8

LLMs as agents

• MemGPT focuses on equipping agents with long-term memory of user inputs, rather than broader interactive-agent capabilities.Related agents combine memory and planning with emergent social behavior (Park et al., 2023), use web-search pagination resembling MemGPT’s context control (Nakano et al., 2021), and improve planning through interleaved chain-of-thought reasoning (Yao et al., 2022; Wei et al., 2022), paralleling MemGPT’s ability to plan aloud during function execution. Liu et al. (2023b) instead introduce interactive-agent benchmarks spanning games, puzzles, and web shopping.Source: §4 · p. 8

• MemGPT processed lengthy documents beyond LLM context limits by paging relevant context in and out of memory.Source: §5 · p. 8
• MemGPT enabled conversational agents to maintain long-term memory, consistency, and evolvability over extended dialogues.Source: §5 · p. 8
• An OS-inspired memory hierarchy and interrupt-based control flow provide the illusion of larger context resources within fixed LLM context windows.Source: §5 · p. 8
• Future directions include domains with massive or unbounded contexts, database or cache memory tiers, and improved control-flow and memory-management policies.Source: §5 · p. 8

6.1.1. MEMGPT INSTRUCTIONS (DMR)

• MemGPT’s dialogue instructions require staying in character and answering from core memory and conversation search, while baselines receive conversation summaries and must abstain when information is insufficient.MemGPT is instructed to make its best guess; baselines must return only an answer or 'NO ANSWER'.Source: §6.1.1 · p. 11

6.1.2. LLM JUDGE (DMR / OPENER)

• An LLM judge compares dialogue answers with gold answers, generously accepting matching topics and returning a brief explanation followed by CORRECT or WRONG.Longer answers can receive credit if they include the gold answer’s referenced topic; failure to recall that topic is marked wrong.Source: §6.1.2 · p. 11

6.1.3. SELF-INSTRUCT DMR DATASET GENERATION

• DMR dataset generation uses MSC conversations and personas to create memory questions answerable only from prior chat logs, not persona summaries.The generation prompt explicitly prohibits questions whose answers can be inferred from persona information.Source: §6.1.3 · pp. 11–12

6.1.4. DOCUMENT ANALYSIS INSTRUCTIONS

• MemGPT’s document-analysis instructions require continued archival search and responses containing both the answer and supporting archival text, treating the year as 2018.The prompt assumes answers always exist in archival memory. Baselines instead use supplied documents and return 'INSUFFICIENT INFORMATION' when those documents lack the answer.Source: §6.1.4 · p. 12

6.1.5. LLM JUDGE (DOCUMENT ANALYSIS)

• The document-analysis judge requires a correct answer and corresponding document text, accepting equivalent wording but rejecting missing evidence or 'INSUFFICIENT INFORMATION'.This evaluation is intended to check that answers derive from provided text rather than model weights.Source: §6.1.5 · pp. 12–13

6.1.6. K/V TASK INSTRUCTIONS

• Nested key–value lookup follows values that are themselves keys until reaching a value verified not to be a key.MemGPT is instructed to continue archival searches until this condition holds. Baselines receive a JSON object whose keys and values are 128-bit UUIDs.Source: §6.1.6 · p. 13