PaperlayerView exact revision on arXiv

Source-linked AI summary

Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory

Lei Liu, Xiaoyan Yang, Yue Shen, Binbin Hu, Zhiqiang Zhang, Jinjie Gu, Guannan Zhang

arXiv:2311.08719v1cs.CL

TL;DR

Long-term memory-augmented LLMs repeatedly recall and reason over history, a process that can produce inconsistent thoughts. TiM stores historical thoughts for recall and updates memory through post-thinking; experiments report benefits on real-world and simulated dialogues.

  • Problem

    Repeatedly recalling and reasoning over long-term history can produce inconsistent thoughts, while LLM agents may struggle to process exceptionally lengthy inputs.

    Source: §1 · pp. 1–2

  • Method

    TiM recalls relevant historical thoughts before response generation, then post-thinks after generation and updates an organized memory of thoughts.

    Source: §1 · p. 2, §3.6 · p. 6

  • Results

    Experiments indicate that TiM enhances performance across diverse topics and Chinese and English, including response correctness and coherence.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TiM is designed to improve response quality and consistency in long-term human-AI interactions.

    Source: §5 · p. 8

    In the clinical application, TiM-LLM is only an auxiliary tool for doctors providing treatment options and medical suggestions.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Memory-augmented Large Language Models (LLMs) have demonstrated remarkable performance in long-term human-machine interactions, which basically relies on iterative recalling and reasoning of history to generate high-quality responses. However, such repeated recall-reason steps easily produce biased thoughts, \textit{i.e.}, inconsistent reasoning results when recalling the same history for different questions. On the contrary, humans can keep thoughts in the memory and recall them without repeated reasoning. Motivated by this human capability, we propose a novel memory mechanism called TiM (Think-in-Memory) that enables LLMs to maintain an evolved memory for storing historical thoughts along the conversation stream. The TiM framework consists of two crucial stages: (1) before generating a response, a LLM agent recalls relevant thoughts from memory, and (2) after generating a response, the LLM agent post-thinks and incorporates both historical and new thoughts to update the memory. Thus, TiM can eliminate the issue of repeated reasoning by saving the post-thinking thoughts as the history. Besides, we formulate the basic principles to organize the thoughts in memory based on the well-established operations, (\textit{i.e.}, insert, forget, and merge operations), allowing for dynamic updates and evolution of the thoughts. Furthermore, we introduce Locality-Sensitive Hashing into TiM to achieve efficient retrieval for the long-term conversations. We conduct qualitative and quantitative experiments on real-world and simulated dialogues covering a wide range of topics, demonstrating that equipping existing LLMs with TiM significantly enhances their performance in generating responses for long-term interactions.

1 INTRODUCTION

TiM addresses repeated reasoning and inconsistent thoughts in long-term memory by storing thoughts for recall and post-thinking new thoughts into an evolving memory. It also uses Locality-Sensitive Hashing for efficient retrieval and organizes memories through insert, forget, and merge operations.

1 INTRODUCTION

  • Long-term inputs remain a limitation for LLM agents, and forgetting past information such as symptoms may hinder accurate clinical diagnosis.Prior approaches include internal memory methods that extend sequence handling and external memory caches that retrieve relevant history.Source: §1 · p. 1
  • Repeatedly recalling and reasoning over conversation history can produce inconsistent reasoning paths and wrong responses, while pairwise similarity retrieval becomes time-consuming in long dialogues.The same earlier turn may be reasoned over again for multiple later questions.Source: §1 · pp. 1–2
  • TiM stores historical thoughts for selective recall, then post-thinks after generating a response to add new thoughts to an evolving memory.Its memory organization supports insert, forget, and merge operations.Source: §1 · p. 2
  • Locality-Sensitive Hashing supports efficient thought insertion and recall, and TiM is designed to work with both closed- and open-source LLMs.The cited examples are ChatGPT and ChatGLM[17].Source: §1 · p. 2
  • Experiments indicate that TiM enhances performance across diverse topics and Chinese and English, including response correctness and coherence.The introduction reports experiments on multi-turn dialogue datasets.Source: §1 · p. 2

2 RELATED WORK

The related work describes broad LLM capabilities alongside a lack of strong long-term memory. Prior long-term memory approaches use external caches, but raw-text storage and pairwise similarity retrieval remain challenges.

2.1 Large Language Models

  • LLMs have been applied to tasks including translation, sentiment analysis, and question answering, but are described as lacking strong long-term memory for lengthy contexts and historical information.The section names both closed-source and open-source model examples.Source: §2.1 · p. 3

2.2 Long-term Memory

  • Long-term memory research includes memory-augmented networks and dialogue datasets, as well as MemoryBank, but the surveyed methods are described as storing raw dialogue and requiring repeated reasoning.The text also identifies pairwise similarity calculations as time-consuming for long-term interactions.Source: §2.2 · p. 3
  • The section characterizes existing long-term memory methods as facing challenges in reliable, adaptable memory for language models.The cited approaches also calculate pairwise similarity to recall relevant information.Source: §2.2 · p. 3

3 METHODOLOGY

The methodology section introduces TiM’s workflow and then details its stages and memory mechanisms, covering storage, organization, and retrieval.

3 METHODOLOGY

  • The section first introduces the overall workflow of the proposed TiM framework.It then describes each stage in detail.Source: §3 · p. 3
  • The methodology describes TiM’s stages, including memory-cache storage and the principles for updating memory.Retrieval for recalling thoughts is also included in the stage descriptions.Source: §3 · p. 3
  • The methodology covers retrieval as a mechanism for recalling thoughts from memory.Retrieval is presented alongside storage and memory organization.Source: §3 · p. 3

3.1 Framework Overview

TiM aims to generate accurate responses to new queries while retaining useful historical information across turns. Its framework combines an LLM agent, a hash-table memory cache, and hash-based mapping, with recall-and-generation followed by post-thinking and updating.

3.1 Framework Overview

  • TiM targets more accurate responses to new queries while remembering contextual information from earlier conversation turns.The framework is intended to retain useful historical information across multiple conversations.Source: §3.1 · pp. 3–4
  • The framework combines a pretrained LLM agent, a continually growing hash-table memory cache whose values are single thoughts, and Locality-Sensitive Hashing for finding and saving relevant thoughts.The agent examples include ChatGPT and ChatGLM [17].Source: §3.1.1 · p. 4

3.1.1 Main Components.

  • In recall and generation, the agent retrieves relevant thoughts to answer without repeatedly reasoning over raw history; after answering, it post-thinks over the query-response pair and inserts new thoughts into memory.These are the framework’s two stages: Recall and Generation, then Post-think and Update.Source: §3.1.2 · p. 4

3.2 Storage for Memory Cache

TiM stores self-generated inductive thoughts as hash-indexed memory entries. Thoughts express relations between entities, while locality-sensitive hashing groups nearby embeddings for retrieval.

3.2 Storage for Memory Cache

  • Each thought is stored as a (hash index, thought) tuple, and similar thoughts are assigned to the same group.The hash table supports quick retrieval and provides an index for updating historical thoughts.Source: §3.2.1 · p. 4, §3.2.2 · p. 4

3.2.1 Thoughts-based System.

  • LLM Agent A uses few-shot in-context prompts to generate inductive thoughts expressing relations between entities.An inductive thought is text satisfying a relation triple (E_h, r_i, E_t).Source: §3.2.1 · p. 4

3.2.2 Hash-based Storage.

  • LSH maps d-dimensional embeddings to hash indices using random projection, giving nearby vectors a higher probability of sharing an index.The random matrix R has size (d, b/2), where b is the number of memory groups.Source: §3.2.2 · p. 4

3.3 Retrieval for Memory Recalling

TiM retrieves relevant thoughts in two stages: LSH narrows the search to a nearby group, then similarity scoring selects the top-k thoughts. Restricting pairwise comparisons to that group makes retrieval more efficient than searching the whole cache.

3.3 Retrieval for Memory Recalling

  • The first retrieval stage embeds a query and uses its LSH index to identify the nearest group of thoughts.The index indicates the group expected to contain similar thoughts.Source: §3.3 · p. 4
  • The second stage scores the query against thoughts in that group and recalls the top-k as relevant history.The recalled thoughts are used to answer the query.Source: §3.3 · p. 4
  • Pairwise similarity is calculated within one group rather than across the whole cache, enabling more efficient retrieval than previous memory mechanisms.Source: §3.3 · pp. 4–5

3.4 Organization for Memory Updating

TiM organizes thoughts within hash-table groups through insert, forget, and merge operations. These operations support dynamic memory updates, including removing unnecessary thoughts and consolidating similar ones.

3.4 Organization for Memory Updating

  • Insert stores new thoughts in memory.Source: §3.4 · p. 5
  • Forget removes unnecessary thoughts, including contradictory thoughts.Source: §3.4 · p. 5
  • Merge combines similar thoughts, such as thoughts with the same head entity.Source: §3.4 · p. 5

3.5 Parameter-efficient Tuning

The paper uses LoRA as a computation-efficient fine-tuning approach for settings with limited computational resources. Experiments use rank 16 and train for 10 epochs.

3.5 Parameter-efficient Tuning

  • LoRA freezes the original weights and optimizes rank-decomposition matrices, reducing the number of trainable parameters.For a linear layer, the paper expresses the update as y = Wx + BAx, with r much smaller than min(d, k).Source: §3.5 · p. 5
  • The experiments set LoRA rank r to 16 and train the LLM models for 10 epochs.Source: §3.5 · p. 5
  • The fine-tuning stage adapts LLMs to multi-turn conversations to provide appropriate and effective responses.Source: §3.5 · p. 5

3.6 Insightful Discussion

TiM stores thoughts rather than raw dialogue, supporting direct recall without repeated reasoning and more flexible memory organization.

3.6 Insightful Discussion

  • Prior memory mechanisms store raw conversation text and require repeated reasoning, while TiM stores thoughts that can be recalled directly.TiM also offers more organization operations than simple read and insert.Source: §3.6 · p. 6
  • TiM is designed as an LLM-agnostic module that can be combined with other language models.Some token-storing prior mechanisms require adapting the LLM architecture.Source: §3.6 · p. 6

4 EXPERIMENT

The experiments evaluate TiM with two open-source LLMs on three dialogue datasets and report stronger response outcomes, faster retrieval, and a medical-agent example.

4.1.1 Dataset.

  • The evaluation uses KdConv, GVD, and a manually constructed medical dataset; RMD tests diagnosis on 80 conversations.KdConv covers film, music, and travel; GVD includes 15 virtual users over 10 days and 194 test questions.Source: §4.1.1 · p. 6

4.1.2 LLM.

  • The experiments use ChatGLM, a 6.2-billion-parameter bilingual model, and Baichuan2, a 13-billion-parameter multilingual model.Both models initially lack long-term memory and specific adaptability to long-term conversations.Source: §4.1.2 · p. 6

4.1.3 Baseline.

  • Baselines include answering without memory and SiliconFriend, which stores raw text and supports reading.Source: §4.1.3 · p. 6

4.1.4 Evaluation Protocol.

  • Human evaluators, blinded to model identity by shuffling predictions, assess retrieval accuracy, response correctness, and contextual coherence.Source: §4.1.4 · p. 6

4.2.1 Results on GVD dataset.

  • TiM outperforms SiliconFriend across English and Chinese GVD metrics, leads across KdConv topics, and improves RMD response correctness and contextual coherence.The comparison table covers three datasets with top-5 thoughts recalled from memory.Source: §4.2.1 · p. 6, §4.2.2 · p. 6, §4.2.3 · p. 6

4.3.1 Retrieval Time.

  • TiM reduces single-retrieval time by about 0.1 ms against pairwise-similarity retrieval, with memory length fixed at 140.Source: §4.3.1 · pp. 6–7, Table 3 · p. 7
  • On KdConv Travel, top-1 retrieval accuracy exceeds 0.7 and top-10 reaches 0.973; overall retrieval accuracy rises as k increases.Top-5 recall can improve existing LLM performance on long-term conversations.Source: Figure 6 · p. 7, §4.3.1 · p. 7

4.4 Industry Application

  • TiM-LLM is an auxiliary clinical tool; the illustrated patient-doctor case contrasts missed prior symptoms without TiM with a more comprehensive assessment using it.The application is shown through a real-world conversation example.Source: §4.4 · pp. 7–8, Figure 7 · p. 8

5 CONCLUSION

TiM stores historical thoughts in an evolving memory, so LLMs can recall relevant thoughts without repeated reasoning. The paper reports significant benefits in dialogue experiments and introduces LSH for efficient retrieval.

5 CONCLUSION

  • TiM recalls relevant historical thoughts before response generation and post-thinks afterward to incorporate historical and new thoughts into evolving memory.The paper says its memory-organization principles support dynamic updates.Source: §5 · p. 8
  • Qualitative and quantitative experiments on real-world and simulated dialogues demonstrate significant benefits from equipping LLMs with TiM.The paper presents TiM as an approach to improve response quality and consistency in long-term human-AI interactions.Source: §5 · p. 8
  • Locality-Sensitive Hashing is introduced into TiM to support efficient retrieval in long-term conversations.Source: §5 · p. 8
Loading 2311.08719v1…