MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems

1. Introduction

• Scaling training data and model parameters yields diminishing gains as high-quality data depletes, leaving continual improvement of LLM systems unresolved.The introduction also notes that inference-time scaling is bounded by backbone models and requires substantial engineering.Source: §1 · p. 1
• Human and traditional AI examples point to memory architecture and feedback utilization as components of continual learning.The discussion connects learning from interaction to developing better strategies for future tasks.Source: §1 · pp. 1–2
• Prior memory benchmarks largely test homogeneous, static long-context tasks and omit procedural memory built from test-time user feedback.The authors argue that this makes those benchmarks unsuitable for evaluating continual learning.Source: §1 · p. 2
• MemoryBench evaluates continual learning by simulating user feedback across datasets with multiple domains, languages, and task formats.It tests whether systems use both task context and feedback logs to improve performance.Source: §1 · p. 2
• Existing LLM memory systems are not effective or efficient enough to continually learn from user feedback or use procedural knowledge to improve performance.The authors call for more advanced parametric and non-parametric algorithms for continual learning.Source: §1 · p. 2

2.1. Preliminary and Taxonomy

• MemoryBench distinguishes declarative factual knowledge, split into semantic and episodic memory, from procedural knowledge about task execution.It classifies feedback as explicit when it reflects output quality and implicit when it arises during service use without judging the response.Source: §2.1 · pp. 2–3
• Its benchmark includes diverse tasks across domains and input-output formats, unlike prior benchmarks focused mainly on long-input, short-output factual reading comprehension.The authors state that MemoryBench provides all types of memory and feedback data.Source: §2.1 · p. 3

2.2. Architecture

• The architecture uses task context and feedback logs as memory resources, with training cases routed to the user simulator and test cases to the performance monitor.The monitor uses dataset metrics and an LLM-as-judge to combine multiple metrics into a task score, then aggregates scores across cases and datasets.Source: §2.2 · pp. 3–4

2.3. Data Preparation

• The benchmark collects 11 public datasets spanning three domains, four task formats, and two languages.Cases are standardized as a query q, evaluation metadata v, and optional task context c.Source: §2.3 · p. 4

2.4. Feedback Simulation

• Feedback simulation combines an LLM-as-user paradigm with a two-stage programmable action simulator.For tasks with verifiable ground truth, objective scores map to feedback templates; for subjective open-ended tasks, an LLM simulates user responses using a persona and evaluation criteria.Source: §2.4 · pp. 4–5
• The simulator generates verbal critiques and follow-up turns, or satisfaction-based actions such as like, dislike, and copy, using task-dependent mapping functions.These action mappings are designed to represent diverse user behavior patterns.Source: §2.4 · p. 5
• Human annotators had difficulty distinguishing simulated verbal feedback from human-written feedback, or had low agreement about which was more human-like.The paper reports this as evidence about the quality of its LLM-as-user feedback generation.Source: §2.4 · p. 5

3.1. Baseline Categorization

• The experiments test BM25 and embedding-based RAG, A-Mem, Mem0, and MemoryOS against Vanilla, which answers queries directly with the backbone LLM.The same Qwen3-8B backbone is used across baselines; the RAG variants organize feedback as session or message documents.Source: §3.1 · pp. 5–6

3.2. Experimental Setup

• Data are grouped into seven domain or task-format partitions, with training feedback kept separate from test cases; evaluation combines dataset metrics using an LLM judge and normalized scores.The experiments include off-policy and on-policy settings, but the main-text results use off-policy learning with explicit verbal feedback.Source: §3.2 · p. 6

3.3. Results and Discussions

• Simulated feedback is reasonably accurate and useful, and memory-supported systems mostly outperform Vanilla, although Vanilla beats all memory-based systems on SiLO.The SiLO result illustrates that memory systems’ effectiveness can vary significantly across task formats.Source: §3.3 · pp. 6–7
• A-Mem, Mem0, and MemoryOS do not consistently outperform RAG; on Locomo, memory methods outperform BM25 RAG, but their generalizability across MemoryBench tasks is limited.RAG uses task context and feedback logs as retrieval material; the comparison also reflects differences from earlier long-context reading-comprehension evaluations.Source: §3.3 · p. 7, Figure 3 · p. 7
• Existing memory mechanisms struggle to handle declarative context and procedural feedback together, while vertical-domain feedback may be harder to interpret because it requires more domain knowledge.The authors identify limited understanding and use of procedural feedback as a constraint on continual learning.Source: §3.3 · pp. 7–8
• Memory and prediction times vary across tasks: MemoryOS construction is mostly longer than 17 seconds per case, while Mem0 inference is abnormally slow on LiLo.Figure 4 compares average per-case memory-operation and test-prediction times across four task-format partitions.Source: §3.3 · p. 7, Figure 4 · p. 8

4. Related Work

• Existing memory-based LLM systems typically combine memory processing, a database, retrieval, and generation to manage information and experience.These modules process conversations into memory entries, store and index them, retrieve relevant entries, and use them with the current task context to generate responses.Source: §4 · p. 8
• MemoryBench is described as the first benchmark to evaluate LLM systems’ continual learning from user-feedback logs, a capability not covered by prior multi-turn conversation studies.Those studies treat utterances as local context and discard them after each task; the passage also notes the lack of released commercial user logs and a comprehensive benchmark.Source: §4 · p. 8
• User simulation has a long history in IR, and recent LLM behavior-analysis and LLM-as-judge approaches make simulated feedback for LLM systems possible.Traditional frameworks often targeted simpler interfaces, such as ranked lists.Source: §4 · p. 8

5. Conclusion

• MemoryBench evaluates whether LLM systems can use service-time user feedback and build procedural memory to improve performance continuously, unlike benchmarks focused only on long input contexts.Feedback is simulated using an LLM-as-user paradigm.Source: §5 · p. 8
• Experiments find existing memory-based LLM systems remain far from effective and efficient use of feedback across tasks, languages, and domains.The authors note that the benchmark’s simulation framework may introduce systematic biases to evaluation.Source: §5 · p. 9
• The authors present MemoryBench as an initial step toward future research on memory and optimization algorithms for LLM systems.They conclude that substantial work remains before LLM systems achieve actual continual learning abilities.Source: §5 · p. 9

Impact Statement

• Systems that adapt from user feedback may reinforce biases or misinterpretations and may expose user privacy during optimization and service.The statement recommends validation, transparency in data use, and oversight mechanisms to mitigate these risks.Source: Impact Statement · p. 9
• MemoryBench’s risks are described as not significant because it uses public LLM datasets and simulated user feedback.Source: Impact Statement · p. 9
• The authors open-source data, preprocessing scripts, and baseline implementations to support reproducible research on adaptive LLM systems.Source: Impact Statement · p. 9

A.1. Data Preparation and Evaluation Protocols

• Text length is measured in tokens with the Qwen3-8B tokenizer, combining query and corpus when available for inputs and using golden-answer lengths for outputs when available.For HelloBench, IdeaBench, and WritingBench, output length is the average generated length from Qwen3-8B.Source: §A.1 · p. 13
• A 600-token boundary distinguishes short from long inputs and outputs.Source: §A.1 · p. 13
• The protocol follows each dataset’s official judge prompts, then uses an LLM judge to combine multiple criteria into a 1-to-10 score and normalizes results within datasets before averaging.WritingBench uses WritingBench-Critic-Model-Qwen-7B4; other datasets use DeepSeek-V3.Source: §A.1 · p. 13

LexEval

• LexEval contributes Chinese legal summary generation, judicial analysis generation, and open-ended QA subsets, evaluated with Rouge-L.The selected subsets are labeled 5-1, 5-2, and 5-4.Source: §A.1.1 · p. 13

JuDGE

• JuDGE combines training and test sets for legal judgment generation, evaluating reasoning, legal references and charges, and sentence and fine amounts.Its metrics include METEOR and BERTScore for reasoning and judgment sections, precision and recall for legal references and charges, and differences in fines and prison sentences.Source: §A.1.1 · p. 13

Writing datasets

• WritingBench contributes Politics & Law, Academic & Engineering, and Creative & Design subsets, while WritingPrompts contributes 2,000 test-set stories evaluated with METEOR.WritingBench evaluates English and Chinese writing with dynamic rules and critic scores from 1 to 10.Source: §A.1.1 · p. 13

Locomo and DialSim

• Locomo treats each dialogue as a corpus and each question as a query, while DialSim uses TV-show scripts and answerable, time-aligned Fan Quiz questions.MemoryBench removes DialSim’s full dialogue history from the prompt so baselines must handle memory automatically; it samples 1,000 queries per subset and uses exact match followed by LLM-as-Judge.Source: §A.1.1 · pp. 13–14

SciTechNews

• SciTechNews uses its test set to assess science journalism with Rouge-L and BERTScore-F1 for factual accuracy and CLI, FKGL, and DCRS for readability.The passage states that lower readability scores indicate articles better suited for science news.Source: §A.1.1 · p. 14

IdeaBench

• IdeaBench evaluates generated research ideas against target-paper ideas using an Insight Score, with BERTScore and LLM-as-Judge idea overlap as reference metrics.The Insight Score ranks ideas against user-specified criteria such as novelty and feasibility, then uses the target paper’s ideas’ relative ranking.Source: §A.1.1 · pp. 14–15

HelloBench

• HelloBench is divided into Academic & Knowledge QA, Academic & Knowledge Writing, and Creative & Design subsets covering selected QA, writing, creative, and text-completion tasks.Evaluation follows the recommended average of checklist-based dimension scores.Source: §A.1.1 · p. 15

Other evaluation datasets

• NF-Cats uses its test set with LLM-as-Judge scoring, while LimitGen-Syn prompts for three paper limitations and selects the highest-scoring generated limitation.LimitGen-Syn specifies the limitation subtype and uses an LLM-based 0–5 rating after coarse-grained subtype matching.Source: §A.1.1 · pp. 13, 15

JuDGE

• JuDGE’s prompt asks for one 1–10 overall judgment-document quality score after reviewing inputs, outputs, ground truth, and the provided metrics.The prompt incorporates penalty, conviction, and referencing accuracy measures, including sentence and fine accuracy, charge precision and recall, and legal-reference and text-similarity scores.Source: §A.1.2 · p. 16

IdeaBench

• IdeaBench’s prompt combines semantic similarity, idea overlap, novelty, and feasibility metrics into one 1–10 overall research-idea quality score.The prompt asks the judge to review the generated idea, ground truth, and pre-calculated metrics.Source: §A.1.2 · p. 17

SciTechNews

• The task asks for a single integer score from 1 to 10 for a generated popular-science article’s overall quality, considering accuracy, readability, coherence, and faithfulness to the source material.The listed metrics include Rouge-L for overlap, BERTScore-F1 for semantic similarity, and CLI for grade-level readability.Source: §A.1.2 · pp. 17–18

Framework overview

• The appendix describes a framework for generating realistic, multifaceted user feedback to evaluate LLMsys continual learning.Its description covers the architecture, LLM-as-User-Simulator, probabilistic user actions, and experiment prompt templates.Source: §A.2 · p. 18

Framework components

• The framework’s described components include an LLM-as-User-Simulator and a probabilistic model for user actions.The appendix also details the framework architecture and prompts used in experiments.Source: §A.2 · p. 18

Experiment prompts

• The appendix details prompt templates used in the experiments as part of the simulation framework.The framework is designed to produce feedback signals for evaluating continual learning capabilities.Source: §A.2 · p. 18

Architecture

• The dual-path architecture returns a session termination signal, natural-language user response, simulated interface action, and numerical satisfaction score.The path is selected according to dataset type, balancing high-fidelity evaluation for complex subjective tasks with efficiency for objective criteria.Source: §A.2.1 · p. 18

Metric-Based Direct Evaluation

• For objective reading-comprehension tasks such as Locomo and DialSim, direct evaluation converts task metrics into satisfaction scores without costly LLM-based evaluation.DialSim maps incorrect and correct answers to scores of 3 and 9; Locomo maps answer F1 to a 1–10 scale, with thresholds including F1 ≥0.9 yielding 10, ≥0.8 yielding 9, and ≥0.5 yielding 6.Source: §A.2.1 · pp. 18–19

LLM-as-User-Simulator

• For nuanced writing, coding, and open-ended QA tasks, an LLM-as-User-Simulator provides contextual feedback as a proxy for a human user.The described example strong simulator is Qwen-3-32B.Source: §A.2.1 · p. 19

• Human-like feedback generation is contingent on a structured and comprehensive prompting system.Source: §A.2.2 · p. 19
• The prompting system has two distinct components: a profile prompt and a test prompt.Source: §A.2.2 · p. 19
• The profile prompt defines the simulated user’s persona, while the test prompt supplies context for a specific evaluation.Source: §A.2.2 · p. 19

• The profile prompt establishes the simulated user’s stable characteristics and behavioral rules, including persona, domain expertise, evaluation criteria, and behavioral constraints.Source: Prompting Strategy · p. 19
• The test prompt assembles the current conversation history, the end user’s underlying goal, and ground-truth evaluation context such as a golden answer.The ground-truth context lets the simulator act as an oracle.Source: Prompting Strategy · p. 19
• The qualitative stage returns JSON with reasoning about response quality and the next natural-language user utterance, if the conversation continues.The simulator is instructed to respond in the conversation’s language and format its output as machine-readable JSON.Source: Prompting Strategy · p. 19
• For Path 2 datasets, the SatisfactionScorer assigns a 1–10 satisfaction score using ground-truth context; Path 1 scores for DialSim and Locomo are generated deterministically.The score evaluates the LLMsys’s last response and is the primary input for modeling subsequent user actions.Source: Prompting Strategy · p. 19

A.2.3. PROBABILISTIC USER ACTION MODELING

• For LLM-scored datasets, scaled sigmoids make liking rise and disliking fall with satisfaction, calibrated to 6.5% feedback and an 86:14 like-to-dislike split (P(L) ≈0.0559, P(D) ≈0.0091).The global feedback rate and split follow Shuster et al. (2022) and Chatterji et al. (2025).Source: §A.2.3 · pp. 19–20
• The general sigmoid uses S0L = 7.5, S0D = 4.5, and kL = kD = 1.5; its parameters are tunable to simulate user populations with different criticality.Scaling constants cL and cD are analytically derived to match the target global probabilities.Source: §A.2.3 · p. 20
• DialSim maps incorrect score 3 to dislikes and correct score 9 to likes, yielding 9.9% and 2.1%; Locomo uses separately calibrated sigmoid constants for its deterministic F1 score distribution.Source: §A.2.3 · p. 20
• Copy probability is four times the like probability for long-output tasks and zero for short-output tasks.Long-output examples include SiLo and LiLo; short-output examples include LiSo and SiSo.Source: §A.2.3 · p. 20

A.2.4. PROMPT TEMPLATES

• The prompt directs simulated users to assess response quality and accuracy against the initial request, decide whether to continue or end, and supply a user response only if continuing; scoring uses a 1–10 scale.The assessment considers correctness and completeness, and output must be valid JSON.Source: §A.2.4 · pp. 21–22

Example: SciTechNews Dataset Prompt

• The SciTechNews specialization evaluates journalistic transformations for accessibility, accuracy to source material, style, engagement, and clarity for general audiences.It frames the simulated user as a science journalist or editor and uses reference popular titles and abstracts as evaluation context.Source: Example: SciTechNews Dataset Prompt · pp. 22–23

A.2.5. HUMAN ANNOTATION OF FEEDBACK GENERATION QUALITY

• Human annotators found simulated feedback difficult to distinguish from human-written feedback in naturalness, relevance, and overall quality.Naturalness agreement was low; half of annotators favored simulated feedback on naturalness, and slightly more than half favored it on relevance and overall quality.Source: §A.2.5 · p. 24

A.3.1. DATASETS MIXTURES DETAILS

• Each dataset category was capped at 250 sampled instances when available, split 4:1 into training and test sets, then combined across datasets.MemoryBench-Full uses all available data but is excluded from experiments because dataset sizes vary substantially and could distort averages.Source: §A.3.1 · p. 24

A.3.3. EXPERIMENT SETTINGS

• Off-policy training generates up to three-turn dialogues with a feedback simulator, loads them into memory for test questions, and evaluates answers using dataset-specific metrics aggregated across datasets.For static-knowledge tasks, relevant knowledge is loaded before interaction; Vanilla receives conversation sessions in its prompt, while Mem0 is excluded from DialSim-theoffice because it could not ingest the corpus in reasonable time.Source: §A.3.3 · pp. 26–27

A.3.4. OFF-POLICY EXPERIMENTS

• Off-policy results use Qwen3-8B and report dataset-aggregated min-max normalization and Z-scores; a Qwen3-32B backbone variant reuses Qwen3-8B-generated training dialogues.Normalization bounds for other experiments are aligned to the Qwen3-8B off-policy results for each dataset.Source: §A.3.4 · pp. 27–28, Table 10 · p. 28, Table 11 · p. 28

A.3.5. FEEDBACK EFFECT EVALUATION

• With feedback, LLMsys score higher on most datasets for the current question, and most memory-enabled systems improve on LoCoMo with binary correctness feedback.The current-question comparison samples the first 100 questions for datasets evaluated by LLM-as-judge; LoCoMo is tested in full using only systems with memory mechanisms.Source: §A.3.5 · p. 28
• LLMsys consistently outperform Vanilla across domain dataset categories overall, but some systems fall below Vanilla when irrelevant memories enter the context.The comparison excludes LoCoMo and DialSim from the Open domain because Vanilla cannot handle their full static knowledge.Source: §A.3.5 · p. 32

A.3.6. ON-POLICY EXPERIMENTS

• On-policy learning repeatedly samples 100 training instances, generates up to three dialogue rounds, adds those dialogues to memory, and evaluates the updated system.Static knowledge is loaded into memory before the steps begin.Source: §A.3.6 · p. 32
• The study’s on-policy plots track aggregated scores across training steps on Open, Academic, and Legal domains, using generation parameters matching the off-policy experiments.Figures 5, 6, and 7 correspond to Open, Academic, and Legal, respectively.Source: §A.3.6 · p. 32, Figure 5 · p. 34, Figure 6 · p. 34

A.3.7. STEPWISE OFF-POLICY EXPERIMENTS

• Stepwise off-policy experiments load previously generated dialogues in batches of 100, evaluate the system on the full test set after each batch, and plot performance across training steps for Open, Academic, and Legal.The reported domain plots are Figures 8, 9, and 10, respectively.Source: §A.3.7 · p. 32, Figure 8 · p. 35

A.3.8. TIME CONSUMPTION ANALYSIS

• The authors identify fixed-format storage as a limitation and call for memory systems that flexibly handle diverse input modalities and hierarchical data.They note that dialogue may need both fine-grained entries for retrieval and a whole-dialogue entry for contextual integrity.Source: §A.3.8 · p. 38

A.3.9. ACTION FEEDBACK

• Like- and copy-feedback experiments selected action-containing training dialogues while retaining the off-policy test set; Legal copy-feedback Mem0 inference was not completed.Mem0 took over 10 hours to memorize the Legal copy-feedback dialogues and then required about 22 minutes per case. The study also trained SFT LoRA adapters on filtered first-turn dialogues as a baseline.Source: §A.3.9 · p. 38, Table 20 · p. 42
• Under noisy like feedback, RAG systems fluctuated little and did better at high S0L, while A-Mem peaked at S0L = 6.5; under Gaussian noise, Embed-S and A-Mem declined as σ rose, while BM25-S and Mem0 were less sensitive.Mem0 had no clear trend across S0L values; with Gaussian perturbations, its performance declined at σ = 0.8 and σ = 1.0 before modestly recovering at σ = 1.2.Source: §A.3.9 · pp. 38–39, Figure 13 · p. 40, Figure 14 · p. 40

A.3.11. CASE STUDY

• In a Legal-domain case study, none of the systems retrieved entries directly addressing the question, but BM25-S retrieved more relevant legal material with fewer disrupting tokens, which may explain its higher Rouge-L.The example also motivates further work on architectures and algorithms that better use different types of context and memory information.Source: §A.3.11 · p. 40, Table 23 · p. 43

A.3.8. TIME CONSUMPTION ANALYSIS

• MemoryOS had the highest memory cost, Mem0 ranked second, and A-Mem’s memory efficiency was close to RAG systems while prediction efficiency remained comparable.For pre-collected conversations, MemoryOS memorized 2,347 DialSim-theoffice sessions in 0.4937 seconds and a 19-session LoCoMo conversation in 0.0125 seconds; Mem0 took over 6 minutes for the LoCoMo conversation and failed to finish DialSim-theoffice. Mem0 also slowed to over 20 minutes per entry and had extremely slow inference in some settings.Source: §A.3.8 · pp. 37–38, Figure 11 · p. 39, Table 19 · p. 41

A.3.10. EFFECT OF BACKBONES FOR LLMSYS AND USER SIMLUATION

• Mistral3.2-24B outperformed Qwen3-8B as the LLMsys backbone, but baseline rankings mostly persisted across LLMsys and user-simulator backbones, with BM25-M best.The reported pattern indicates that differences between LLMsys largely reflect system design rather than the LLMsys or user-simulator backbones.Source: §A.3.10 · pp. 39–40, Table 22 · p. 42