Source-linked AI summary
MemoryBank: Enhancing Large Language Models with Long-Term Memory
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, Yanlin Wang
TL;DR
LLMs lack long-term memory for sustained interactions, motivating a mechanism that can retain, retrieve, and adapt memories over time. The paper introduces MemoryBank, including Ebbinghaus-inspired updating, and demonstrates it in the psychologically tuned SiliconFriend companion. Reported evaluations indicate memory recall, user-behavior understanding, empathetic support, and applicability across multiple model and language settings.
Problem
LLMs lack long-term memory, limiting sustained interactions such as AI companionship and psychological counseling.
Method
MemoryBank combines storage, retrieval, updating, event summarization, and personality modeling, and is integrated into SiliconFriend with psychological-dialog tuning.
Results
MemoryBank-equipped SiliconFriend recalls memories, deepens understanding of user behavior, and provides empathetic emotional support in qualitative and quantitative evaluations.
Takeaways & Limitations
MemoryBank is presented as applicable to open- and closed-source models, Chinese and English, and configurations with or without memory forgetting.
Takeaways & Limitations
The memory updating model is exploratory and highly simplified, while real memory varies across people and information types.
Abstract
from arXiv · showhide
Revolutionary advancements in Large Language Models have drastically reshaped our interactions with artificial intelligence systems. Despite this, a notable hindrance remains-the deficiency of a long-term memory mechanism within these models. This shortfall becomes increasingly evident in situations demanding sustained interaction, such as personal companion systems and psychological counseling. Therefore, we propose MemoryBank, a novel memory mechanism tailored for LLMs. MemoryBank enables the models to summon relevant memories, continually evolve through continuous memory updates, comprehend, and adapt to a user personality by synthesizing information from past interactions. To mimic anthropomorphic behaviors and selectively preserve memory, MemoryBank incorporates a memory updating mechanism, inspired by the Ebbinghaus Forgetting Curve theory, which permits the AI to forget and reinforce memory based on time elapsed and the relative significance of the memory, thereby offering a human-like memory mechanism. MemoryBank is versatile in accommodating both closed-source models like ChatGPT and open-source models like ChatGLM. We exemplify application of MemoryBank through the creation of an LLM-based chatbot named SiliconFriend in a long-term AI Companion scenario. Further tuned with psychological dialogs, SiliconFriend displays heightened empathy in its interactions. Experiment involves both qualitative analysis with real-world user dialogs and quantitative analysis with simulated dialogs. In the latter, ChatGPT acts as users with diverse characteristics and generates long-term dialog contexts covering a wide array of topics. The results of our analysis reveal that SiliconFriend, equipped with MemoryBank, exhibits a strong capability for long-term companionship as it can provide emphatic response, recall relevant memories and understand user personality.
1 Introduction
MemoryBank addresses LLMs’ lack of long-term memory by retaining and retrieving interaction history, updating memories, and modeling user personalities. SiliconFriend demonstrates this approach for long-term companionship through psychological-dialog tuning and qualitative and simulated-dialog evaluations.
- Motivation: LLMs lack long-term memory needed for sustained interactions such as personal companionship, psychological counseling, and secretarial tasks.Long-term memory supports contextual understanding, meaningful interaction, and understanding user behavior over time.
- MemoryBank: MemoryBank combines memory storage, retrieval, updating, event summarization, and user-personality modeling to support long-term interaction.Its updating mechanism is inspired by the Ebbinghaus Forgetting Curve and selectively preserves, forgets, or strengthens memories based on elapsed time and significance.
- Generalizability: MemoryBank accommodates both closed-source models such as ChatGPT and open-source models such as ChatGLM or BELLE.
- SiliconFriend: SiliconFriend is an LLM-based companion integrated with MemoryBank and tuned with 38k psychological conversations to support empathetic and careful responses.
- Evaluation: The evaluation combines qualitative analysis of real-world user dialogs with quantitative analysis using simulated dialogs from 15 virtual users across 10 days and 194 probing questions.ChatGPT generates dialog contexts for users with diverse personalities and topics.
- Contributions: The paper reports MemoryBank’s generalizability across model availability, Chinese-English bilingual use, and settings with or without memory forgetting.The reported contributions also include SiliconFriend’s memory recall, empathetic companionship, and understanding of user behaviors.
2 MemoryBank: A Novel Memory Mechanism Tailored for LLMs
MemoryBank organizes long-term information in storage, retrieves contextually relevant memories, and updates retention using a simplified Ebbinghaus-inspired mechanism. Its design includes detailed and summarized interaction records, evolving personality assessments, dense retrieval, and time-sensitive forgetting.
- Architecture: MemoryBank is structured around memory storage, context-specific memory retrieval, and an Ebbinghaus-inspired memory updater.
- Memory Storage: Memory storage records chronological, timestamped conversations, daily event summaries, global summaries, and evolving assessments of user personalities.The hierarchy distills verbose dialogues into daily and global summaries while personality insights are continuously updated and aggregated.
- Memory Retrieval: Memory retrieval treats conversation turns and event summaries as memory pieces, encodes them into vectors, and indexes them with FAISS for relevance search.The current conversation is encoded as a query, and the encoder can be replaced with any suitable model.
- Memory Updating: Memory updating is intended to make long-term interaction behavior more human-like by forgetting old, infrequently recalled, less-important memories.
- Memory Updating: R = e^-t/S models retention using elapsed time t and memory strength S, which increases by 1 and resets t to 0 whenever a memory is recalled.The update makes recalled memories persist longer and lowers their forgetting probability.
- Limitation: The memory updating model is explicitly exploratory and highly simplified because real memory varies across people and information types.
3 SiliconFriend: An AI Chatbot Companion Powered by MemoryBank
SiliconFriend combines psychological-dialogue tuning with MemoryBank to provide an empathetic companion that recalls interactions and adapts to users. The implementation supports multiple LLM backbones and uses memory retrieval and Ebbinghaus-inspired updating for personalized responses.
- SiliconFriend is an emotional companion designed to recall pertinent memories and understand users’ personalities and emotional states.
- SiliconFriend accommodates ChatGPT, ChatGLM, and BELLE, including both closed-source and open-source model backbones.
- Its two-stage development tunes open-source LLMs with 38k psychological dialogues, then integrates MemoryBank for storing and retrieving interactions and user portraits.
- LoRA provides computation-efficient tuning by learning low-rank matrices while keeping original model weights frozen.
- MemoryBank updates stored memories using principles inspired by the Ebbinghaus Forgetting Curve, allowing memory persistence to reflect recall and elapsed time.
- The system retrieves relevant memories with language-specific embeddings and organizes them with user portraits and event summaries in the conversation prompt.
- The resulting chatbot is presented as capable of remembering and learning from past interactions to provide personalized and empathetic experiences.
4 Experiments
The experiments evaluate SiliconFriend through real-world qualitative examples and larger-scale simulated quantitative tests. Results indicate empathetic support, successful memory recall, personality-tailored interaction, and strong performance across model variants and languages.
- The evaluation combines qualitative analysis of real-time user conversations with quantitative analysis of simulated conversations and memory probing questions.
- Psychological Companionship: SiliconFriend provides empathetic responses and constructive suggestions for emotional difficulties, contrasting with baseline ChatGLM responses.
- Memory Recall Analysis: After several days, SiliconFriend recalled previously suggested books and algorithms and identified an event not previously discussed in the dialogue.
- Personality Interaction Analysis: SiliconFriend recommended activities tailored to users’ interests based on diverse personality traits.
- Quantitative Analysis: The quantitative evaluation used 10 days of conversations involving 15 virtual users and 194 probing questions in English and Chinese.
- Quantitative Analysis: The evaluation measured memory retrieval accuracy, response correctness, and contextual coherence.
- Quantitative Analysis: SiliconFriend ChatGPT performed highly across all metrics, while BELLE and ChatGLM also showed high retrieval accuracy across model backbones.
- Quantitative Analysis: ChatGLM and ChatGPT performed better in English, whereas BELLE performed better in Chinese.
5 Related Works
Prior work expanded neural-model memory and studied long-range conversations, but reliable, adaptable long-term memory for LLMs remains unresolved. Existing conversation datasets also fall short of the extended interactions and user-portrait capabilities needed for long-term AI companions.
- Memory-augmented networks interact with external memory matrices to maintain and manipulate information over extended periods.
- Existing memory approaches have not fully addressed reliable, adaptable long-term memory for LLMs.
- Long-range conversation datasets include multi-session human-human chats, but are generally restricted to a few conversation rounds.
- The paper identifies a remaining need for long-term memory mechanisms that support personalized and persistent LLM interactions.
6 Conclusion
MemoryBank addresses LLM memory limitations by maintaining context, recalling relevant information, and modeling user personality through an Ebbinghaus-inspired update mechanism. Its application in SiliconFriend and evaluations support personalized, empathetic long-term companionship across model types.
- MemoryBank maintains context, recalls relevant information, and supports understanding of user personality over time.
- Its memory updater draws inspiration from the Ebbinghaus Forgetting Curve to model retention and forgetting over time.
- MemoryBank accommodates open-source models including ChatGLM and BELLE, as well as the closed-source model ChatGPT.
- SiliconFriend applies MemoryBank in a long-term AI companion chatbot and uses psychological dialogue data to provide empathetic emotional support.
- Qualitative and quantitative experiments report memory recall, deeper understanding of user behaviors, and higher-quality empathetic companionship for SiliconFriend.