PaperlayerView exact revision on arXiv

Source-linked AI summary

MemGPT: Towards LLMs as Operating Systems

Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez

arXiv:2310.08560v2cs.AI

TL;DR

Limited context windows hinder extended conversations and long-document analysis, while directly scaling transformer context is computationally costly. MemGPT uses OS-inspired virtual context management to page information between the LLM’s fixed context and external storage. Across document analysis and conversational-agent tasks, it processes lengthy texts and supports long-term memory, consistency, and evolvability.

  • Problem

    Limited context windows hinder LLM performance on extended conversations and long-document analysis, while direct context scaling incurs quadratic computational and memory costs.

  • Method

    MemGPT uses an OS-inspired memory hierarchy and function calls to move information between fixed main context and external storage.

  • Results

    MemGPT out-performed existing LLM-based approaches in document analysis and conversational-agent evaluations constrained by finite context.

  • Takeaways & Limitations

    MemGPT supports processing lengthy texts and maintaining long-term memory, consistency, and evolvability over extended dialogues.

  • Takeaways & Limitations

    In multi-document question answering, MemGPT often stops paging through retriever results before exhausting the retriever database.

Abstract

from arXiv · show

Large language models (LLMs) have revolutionized AI, but are constrained by limited context windows, hindering their utility in tasks like extended conversations and document analysis. To enable using context beyond limited context windows, we propose virtual context management, a technique drawing inspiration from hierarchical memory systems in traditional operating systems that provide the appearance of large memory resources through data movement between fast and slow memory. Using this technique, we introduce MemGPT (Memory-GPT), a system that intelligently manages different memory tiers in order to effectively provide extended context within the LLM's limited context window, and utilizes interrupts to manage control flow between itself and the user. We evaluate our OS-inspired design in two domains where the limited context windows of modern LLMs severely handicaps their performance: document analysis, where MemGPT is able to analyze large documents that far exceed the underlying LLM's context window, and multi-session chat, where MemGPT can create conversational agents that remember, reflect, and evolve dynamically through long-term interactions with their users. We release MemGPT code and data for our experiments at https://memgpt.ai.

1. Introduction

MemGPT addresses fixed context windows by managing information between in-context and external storage, using an OS-inspired virtual-memory design. Evaluations target document analysis and conversational agents, where finite context limits performance.

  • Fixed-length context windows limit LLMs to short conversations and documents.
  • 8?

2. MemGPT (MemoryGPT)

MemGPT extends a fixed-context LLM with OS-inspired hierarchical memory and function-based control over data movement, retrieval, and context overflow. Its processor, queue manager, and event system coordinate memory updates, function execution, and interaction with users.

  • Memory architecture: MemGPT separates main context from external context, requiring out-of-context data to be explicitly moved into the LLM’s fixed window for inference.Main context is analogous to physical memory, while external context is analogous to disk storage.
  • Memory architecture: Prompt tokens comprise read-only system instructions, a fixed-size writable working context, and a FIFO queue for recent messages.Working context can be modified only through MemGPT function calls.
  • Queue management: The queue manager stores incoming messages and LLM outputs in recall storage, retrieves messages through function calls, and reinserts them into the context window.This separates durable conversation storage from the tokens currently supplied to the processor.
  • Queue management: When prompt tokens approach the underlying model’s warning threshold, MemGPT issues a memory-pressure warning so the LLM can preserve important FIFO contents in working or archival storage.The queue manager controls context overflow through an eviction policy, with 70% given as an example warning threshold.
  • Self-directed control: The LLM generates function calls that MemGPT parses, validates, executes, and feeds back into the processor, enabling self-directed memory editing and retrieval.Functions can move data between main and external context, while runtime errors are returned through the feedback loop.
  • Control flow: Events—including user messages, system warnings, user interactions, and timed triggers—can initiate inference, while function chaining supports sequential multi-step operations before control returns to the user.The design therefore supports both user-driven and unprompted processing, as well as multi-function workflows.

3. Experiments

MemGPT is evaluated on long-context conversational and document-analysis tasks, including memory retrieval, engagement, document QA, and nested key-value lookup. Across these experiments, its memory and iterative retrieval improve performance beyond fixed-context baselines, while performance depends on the underlying model and retrieval behavior.

  • 3.1. MemGPT for conversational agents: MemGPT is evaluated on conversational consistency and engagement using multi-session chat data and newly introduced long-context dialogue tasks.The experiments assess whether agents retain knowledge across conversations and use memory to personalize responses.
  • 3.1.1. Deep memory retrieval task (consistency): MemGPT significantly outperforms fixed-context baselines on deep memory retrieval questions about prior sessions.The task asks agents to answer specific questions about topics discussed in sessions 1–5, scoring responses against gold answers.
  • 3.1.2. Conversation opener task (engagement): MemGPT produces engaging conversation openers that perform similarly to and occasionally exceed hand-written human openers.Its openers tend to be more verbose and cover more persona information, with working-context storage identified as important for engagement.
  • 3.2.1. Multi-document question-answering: MemGPT scales document QA beyond the fixed context window by repeatedly querying archival storage, whereas truncation reduces accuracy as relevant document snippets are omitted.Fixed-context baselines are capped by the retriever and available context, while MemGPT can iteratively page through retrieved documents.
  • 3.2.2. Nested key-value retrieval (KV): In nested key-value retrieval, MemGPT with GPT-4 is unaffected by nesting depth, while GPT-3.5 and GPT-4 baselines reach 0 percent accuracy at 1 and 3 nesting levels, respectively.MemGPT performs repeated function-query lookups to combine information across multiple key-value sources; weaker MemGPT configurations decline when they fail to perform enough lookups.

4. Related Work

MemGPT builds on long-context LLMs, retrieval-augmented models, and agent systems with memory and planning capabilities. Its main contribution is a hierarchical, tiered memory architecture that uses long-context models as main memory.

  • Long-context LLMs: Long-context research improves LLM context through sparse attention, low-rank approximations, neural memory, and extending models beyond their training length.
  • Long-context LLMs: MemGPT builds on long-context improvements by using them to increase the size of its main memory.
  • Retrieval-Augmented Models: MemGPT’s external memory draws on retrieval-augmented models that supply LLMs with relevant inputs from external retrievers.
  • LLMs as agents: Prior agent research augments LLMs with memory, planning, and capabilities for acting in interactive environments.

5. Conclusion

MemGPT manages fixed context windows through an operating-system-inspired memory hierarchy and control flow. The system was evaluated for lengthy document analysis and long-term conversational agents, where it could page relevant context and maintain extended interactions.

  • MemGPT manages LLM context windows with a memory hierarchy and control flow analogous to traditional operating systems.
  • In document analysis, MemGPT processed texts beyond current LLM context limits by paging relevant context in and out of memory.
  • In conversational agents, MemGPT maintained long-term memory, consistency, and evolvability over extended dialogues.
  • The paper identifies future directions including other massive-context domains, different memory-tier technologies, and improved control flow.

6. Appendix

The appendix documents prompts and evaluation procedures for MemGPT and its baselines across conversational memory, document analysis, and nested key-value retrieval tasks.

  • Chat and dialogue instructions: MemGPT’s chat persona instructs the agent to remain in role and answer using core memory and conversation search.
  • Chat and dialogue instructions: The dialogue-memory evaluation generates questions that require information from prior chats rather than persona summaries.
  • Document analysis instructions: Document-analysis prompts require searching archival memory and returning both an answer and its supporting document text.
  • Evaluation: An LLM judge scores document-analysis answers as correct when they contain both the correct answer and corresponding document text.
  • Nested key-value retrieval: The nested key-value instructions require recursively returning a value when an associated value is itself another key.
Loading 2310.08560v2…