PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Models

Zecheng Tang, Baibei Ji, Ruoxi Sun, Haitian Wang, WangJie You, Zhang Yijun, Wenpeng Zhu, Ji Qi, Juntao Li, Min Zhang

arXiv:2601.11969v2cs.CLcs.AI

TL;DR

Reward models are needed to automatically evaluate long-term memory management, but scalable RM-based assessment remains underexplored. MemoryRewardBench benchmarks this capability across diverse tasks and settings, finding narrowed open-source–proprietary gaps alongside generational advantages and persistent limitations.

  • Problem

    Scalable, automated reward-model evaluation of long-term LLM memory management remains largely unexplored despite the importance of intermediate memory quality.

  • Method

    MemoryRewardBench benchmarks RMs on comprehension and generation tasks across 10 memory-management configurations, separating outcome-based from process-based judgments.

  • Results

    Open-source RMs largely close the gap with proprietary models on long-context reasoning, while proprietary models remain stronger on long-range-dependency tasks; newer generations outperform predecessors regardless of parameter count.

  • Takeaways & Limitations

    The benchmark exposes current RM capabilities and limitations across diverse memory-management settings and offers guidance for improving reward modeling and memory-centric LLMs.

  • Takeaways & Limitations

    Main experiments randomly order chosen and rejected samples to mitigate positional bias, so conclusions rely on this evaluation design assumption.

Abstract

from arXiv · show

Existing works increasingly adopt memory-centric mechanisms to process long contexts in a segment manner, and effective memory management is one of the key capabilities that enables large language models to effectively propagate information across the entire sequence. Therefore, leveraging reward models (RMs) to automatically and reliably evaluate memory quality is critical. In this work, we introduce MemoryRewardBench, the first benchmark to systematically study the ability of RMs to evaluate long-term memory management processes. MemoryRewardBench covers both long-context comprehension and long-form generation tasks, featuring 10 distinct settings with different memory management patterns, with context length ranging from 8K to 128K tokens. Evaluations on 13 cutting-edge RMs indicate a diminishing performance gap between open-source and proprietary models, with newer-generation models consistently outperforming their predecessors regardless of parameter count. We further expose the capabilities and fundamental limitations of current RMs in evaluating LLM memory management across diverse settings.

1 Introduction

Long-sequence processing can be holistic or segmented, with segmented systems relying on intermediate memory to connect past and present information. MemoryRewardBench benchmarks whether reward models can evaluate that memory-management process across diverse tasks and criteria.

  • Motivation: Segmented processing handles long inputs in chunks while maintaining fixed-size memory that summarizes historical information and integrates newly processed content.This offers an efficient alternative for scalable multi-turn interactions, but makes memory management critical to model success.
  • Benchmark contribution: MemoryRewardBench is introduced as the first benchmark focused on how effectively reward models judge long-term intermediate-memory quality.It shifts evaluation from memory outputs alone to the reward models’ capacity to supervise memory management.
  • Benchmark contribution: The benchmark covers comprehension and generation tasks across 10 memory-management configurations, with contexts ranging from 8K to 128K tokens.Its comparison with existing benchmarks emphasizes broader task and memory-ability coverage.
  • Evaluation criteria: Outcome-based evaluation checks whether a trajectory leads to the correct result, whereas process-based evaluation favors accurate, concise, and logically coherent updates even when outcomes are correct.These criteria separate memory-management quality from final-answer correctness.
  • Main findings: Evaluations of 13 reward models show that performance does not scale monotonically with parameter count, while newer generations consistently outperform predecessors.For example, Qwen3-4B surpasses the substantially larger Qwen2.5-7B-Instruct.

2 Related Work

Prior memory benchmarks mainly assess the memories produced by language models through probing, stress scenarios, or interaction-based evaluation. MemoryRewardBench addresses the comparatively unexplored problem of scalable reward-model-based assessment.

  • Existing memory evaluation: Existing memory benchmarks evaluate retention, updating, regulation, causal coherence, sequential reasoning, or context fidelity across structured and realistic interaction settings.Examples include intermediate-state probing, narrative and domain-specific tasks, memory stress, fact accumulation, and large-context fidelity.
  • Research gap: Despite broad coverage, prior approaches rely heavily on rule-based heuristics or manual annotation rather than automated reward-model assessment.This leaves scalable RM-based evaluation of memory management largely unexplored.
  • Reward models: Reward models act as proxies for human-derived preferences and can be discriminative, generative, or implicit.The paper focuses primarily on generative reward models because this paradigm potentially supports memory evaluation.

3 Introduce MemoryRewardBench

MemoryRewardBench formalizes memory management through sequential, parallel, and mixed patterns, then constructs evaluation tasks and preference pairs for long-context reasoning, dialogue understanding, and long-form generation. Its criteria distinguish final-outcome correctness from the quality and constraint adherence of intermediate memories.

  • Memory management patterns: Sequential memory updates evolve chunk by chunk, with each state combining the previous memory and the current chunk before producing the final outcome.The final result is obtained from the last memory state.
  • Memory management patterns: Parallelism partitions context into independent groups, processes each group separately, and fuses the resulting memory states into an outcome.Mixed management composes parallel processing with subsequent sequential updates.
  • Task overview: The benchmark includes long-context reasoning, multi-turn dialogue understanding, and long-form generation, each requiring memory across extended inputs or multiple generation steps.Dialogue requires persistent temporal memory, while generation requires intermediate states to preserve explicit constraints.
  • Evaluation criteria: Reward-model criteria assess outcome accuracy and process quality for comprehension tasks, while generation tasks emphasize compliance with instruction constraints.Process quality includes concision and relevance of intermediate memory.
  • Benchmark construction: Long-context reasoning pairs correct trajectories with versions containing redundant information or dropped critical information.These NOISE and DROP perturbations create rejected memory-management samples.
  • Benchmark construction: Dialogue pairs use skipped memory updates as rejected samples, including cases where the final response remains correct despite suboptimal intermediate memory.The construction uses A-Mem and Mem0 memory-management methods.

4 Evaluation

The evaluation compares 13 reward models across long-context reasoning, multi-turn dialogue, and long-form generation, revealing differences between proprietary and open-source models and across task types.

  • 13 reward models are evaluated, including 3 proprietary and 10 open-source models, with judgment accuracy reported across benchmark tasks.All evaluated models support context windows of at least 128K tokens.
  • Proprietary vs. Open-source RMs: 74.75 is Claude-Opus-4.5’s highest average score, while GLM4.5-106A12B is the strongest open-source model at 68.21.GLM4.5-106A12B also surpasses proprietary Qwen3-Max, which scores 67.79.
  • Open-source RMs Analysis: Newer-generation models consistently outperform predecessors regardless of parameter count, indicating that practical capability does not monotonically scale with model size.Examples include Qwen3-8B scoring 57.33 versus Qwen2.5-7B at 38.17, and Qwen3-32B outperforming Llama3.3-70B.
  • Cross-Task Capability Characterization: Multi-turn dialogue is the most challenging task, long-form generation is moderately difficult, and long-context reasoning achieves the highest overall scores.Dialogue requires tracking conversational state transitions, while generation requires sustained adherence to global constraints.

5 Ablation Study

The ablations examine memory-management patterns, reward criteria, constraint density, trajectory length, and auxiliary signals, exposing systematic strengths and limitations in reward-model evaluation.

  • 5.1 Effect of Memory Management Patterns: Sequential memory management yields significantly higher reward-model accuracy than parallel processing on long-context reasoning and long-form generation.The results suggest current reward models favor progressive, step-by-step reasoning over parallel processing followed by merging.
  • 5.2 Effect of RM Evaluation Criteria: Reward-model consistency is evaluated by swapping chosen and rejected sample positions, testing whether preferences remain aligned with ground truth.The chosen-first and rejected-first orderings operationalize this position-swapping comparison.
  • 5.2 Effect of RM Evaluation Criteria: Reward-model performance peaks at approximately 25% constraint density, then plateaus or declines as instructions become denser.This indicates only partial ability to use dense, multifaceted constraints when assessing memory fidelity.
  • 5.3 Effect of Memory Management Trajectory Length: Most reward models maintain accuracy above 50% up to 64K tokens, but only GLM4.5-Air and Qwen2.5-72B-Instruct remain stable across most context lengths.Other models show pronounced inconsistency beyond 32K tokens, including severe collapse for Llama models at 64K and 128K.
  • 5.4 Impact of Memory-Enhancement Strategies: Auxiliary semantic tags on memory updates consistently improve reward-model accuracy in multi-turn dialogue evaluation.The comparison uses explicitly tagged versus untagged memory updates.

6 Conclusion

MemRewardBench systematically evaluates whether reward models can assess long-term memory management, showing narrowed open-source gaps on reasoning but persistent weaknesses on long-range dependencies.

  • MemRewardBench is presented as the first benchmark to systematically assess reward models’ evaluation of long-term memory management.It is intended to support practical guidance for improving reward modeling and memory-centric language models.
  • Open-source reward models largely close the gap with proprietary models on long-context reasoning but still lag on multi-turn dialogue and memory-intensive long-form generation.These latter tasks involve long-range dependencies.

A Comparison between LongRewardBench and Existing Memory Benchmarks

LongRewardBench differs from existing memory benchmarks by evaluating reward models rather than only LLM outputs, emphasizing intermediate memory trajectories across diverse abilities and settings.

  • LongRewardBench evaluates reward models’ judgments of long-term intermediate memories, whereas most existing benchmarks directly evaluate LLM memory capabilities through output correctness or consistency.
  • Process-level evaluation distinguishes benchmarks that assess intermediate reasoning or memory updates from those that measure only final-outcome correctness.
  • The benchmark includes both static and dynamic scenarios, covering fixed information as well as continual memory updates, revisions, and overwrites.
  • LongRewardBench covers a broader range of long contexts, enabling more systematic analysis of reward models’ long-horizon memory evaluation capabilities.
  • Its unified framework maps tasks to dialogue understanding, multi-hop reasoning, knowledge update, temporal reasoning, and generation abilities.

B Benchmark Construction

The benchmark constructs preference data from long-context reasoning, dialogue understanding, and long-form generation tasks by contrasting memory trajectories with different management quality.

  • MemRewardBench construction statistics are summarized in Table 4, with task data drawn from BABILong, LongMiT, LoCoMo, MemoryAgentBench, LongProc, LongEval, and LongGenBench.
  • Long-context Reasoning: Long-context reasoning uses key contexts from LongMiT and clue or needle documents from BABILong, processed through chunked MemAgent memory updates.
  • Long-context Reasoning: NOISE introduces incorrect or repeated memory updates, while DROP removes key information to create rejected trajectories lacking crucial evidence.
  • Multi-turn Dialogue Understanding: Dialogue samples pair complete memory chains leading to correct answers with incomplete chains created by skipping memory updates, including Mem and Out negative categories.
  • Long-form Generation: Long-form generation models progressive constraint satisfaction using LongProc, LongGenBench, and LongEval, with sequential generation required for LongProc.

C Evaluation Settings

Evaluation samples use task-specific system prompts and preference pairs whose chosen and rejected trajectories are randomly shuffled before evaluation.

  • Long-context reasoning and multi-turn dialogue use an understanding system prompt, while long-form generation uses a generation system prompt.
  • The chosen and rejected trajectories in each evaluation preference pair are randomly shuffled before being presented for evaluation.

C.2 Evaluation Framework

The evaluation compares sequential and parallel memory-management patterns across long-context reasoning and long-form generation, using standardized sampling settings across models.

  • Evaluations use official APIs for proprietary models and the LOOM-Scope framework for open-source models, with shared sampling settings across all models.
  • The comparison distinguishes Sequential and Parallel patterns, with LR denoting Long-context Reasoning and LG denoting Long-form Generation.
  • The weighted-average results indicate that the sequential pattern is easier than the parallel pattern.

D.2 Failure Case Analysis of Large-scale LLMs

The analysis examines how large language models and reward models behave under varying trajectory lengths and constraint densities. It highlights improved reasoning and instruction adherence in Qwen3-14B, while showing that longer trajectories reduce reward-evaluation accuracy and consistency.

  • Failure Case Analysis: Qwen3-14B more accurately identifies constraint violations and adheres to requirements than the compared baseline models.Its post-training reasoning process balances instruction fidelity against descriptive quality.
  • Trajectory Length: Performance invariably declines as memory-management trajectory length increases.The benchmark reports this trend in its length-based statistical evaluation.
  • Trajectory Length: Longer trajectories also expose inconsistencies in reward-model judgments when the contextual order of chosen and rejected responses is reversed.Additional experiments examine this length-induced bias across tasks.
  • Constraint Density: Constraint-density experiments compare LongEval, LongGenBench, and LongProc under a unified control strategy.LongProc varies known-route availability, while LongEval and LongGenBench vary subsets of predefined constraints.

D.5 Multi-turn Understanding with Auxiliary Signals

The section studies auxiliary signals in multi-turn memory understanding, especially structured tags that organize dialogue and memory processes. Results indicate that semantic tags improve performance and that their removal impairs memory-process discrimination.

  • Auxiliary Signals: Semantic tags help A-Mem summarize dialogue content and provide structured metadata for memory organization.They also support semantic indexing and help locate key information in long conversations.
  • Auxiliary Signals: Meaningful semantic tags significantly improve performance on the memsys task by helping models distinguish memory processes and dialogue segments.The comparison is reported in the structured-tag evaluation.
  • Auxiliary Signals: Removing tags impairs the model’s ability to distinguish memory processes and causes a performance decline.The result supports designing memory-organization mechanisms that preserve semantic structure.
  • Evaluation Cases: The examples span long-context reasoning and long-form generation cases involving irrelevant information, dropped content, and sequential or parallel memory updates.The figures and examples illustrate chosen and rejected memory excerpts and update patterns across question types.
  • Evaluation Prompt: The evaluation prompt asks reward models to compare two step-by-step memory-update trajectories and select the better workflow.Criteria include constraint adherence and preservation of critical information.
Loading 2601.11969v2…