PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues

Chuanrui Hu, Tong Li, Xingze Gao, Hongda Chen, Yi Bai, Dannong Xu, Tianwei Lin, Xiaohong Li, Yunyun Han, Jian Pei, Yafeng Deng

arXiv:2602.01313v3cs.CLcs.AI

TL;DR

Existing conversational-memory benchmarks rarely evaluate the multi-party, evolving, role-shaped interactions found in practical deployments. EverMemBench introduces a benchmark built from dense collaborative dialogues and evaluates recall, memory awareness, and profile understanding, revealing structural failures in attribution, temporal reasoning, and retrieval.

  • Problem

    Current benchmarks largely focus on dyadic or single-topic dialogues, leaving realistic collaborative memory across participants, groups, time, and roles insufficiently evaluated.

  • Method

    EverMemBench uses multi-party, multi-group dialogues with cross-topic interleaving, revisited decisions, role-conditioned personas, and dynamic knowledge updates to evaluate three collaborative-memory dimensions.

  • Results

    Current systems show persistent limitations across all three dimensions, including 26.51% Multi-hop accuracy for Gemini-3-Flash and retrieval failures that miss inferentially relevant information.

  • Takeaways & Limitations

    EverMemBench provides a diagnostic foundation for developing memory systems that represent versioned state, episodic boundaries, cross-group dependencies, and inferential relevance.

  • Takeaways & Limitations

    The benchmark construction uses blueprint specifications to constrain factual content and a blind test to guard against generator-family bias.

Abstract

from arXiv · show

Long-term conversational memory in practical LLM applications is inherently collaborative: information is produced by multiple participants, scattered across groups and channels, revised over time, and implicitly grounded in roles and social context. Yet there is currently no established benchmark that evaluates memory under interaction patterns resembling real-world deployment, as existing benchmarks largely focus on dyadic or single-topic dialogues. In this paper, we introduce EverMemBench, the first benchmark designed for long-horizon collaborative memory, built from multi-party, multi-group conversations spanning over one million tokens with dense cross-topic interleaving, temporally evolving decisions, and role-conditioned personas. EverMemBench evaluates memory systems using 2400 QA pairs across three dimensions essential for real applications: fine-grained recall, memory awareness, and user profile understanding. Our evaluation reveals fundamental limitations of current systems: multi-hop reasoning collapses under multi-party attribution even with oracle evidence (26% accuracy), temporal reasoning fails without explicit version semantics beyond timestamps, and memory awareness is bottlenecked by retrieval, as similarity-based methods miss implicitly relevant information. EverMemBench thus represents a concrete step toward realistic evaluation of LLM memory and a cornerstone benchmark for developing next-generation LLMs that reason over time, roles, and collaborative interaction structure. Our benchmark and code are publicly available at https://github.com/EverMind-AI/EverMemBench.

1 Introduction

EverMemBench addresses the lack of realistic evaluation for collaborative conversational memory by modeling multi-party, multi-group interactions with distributed, evolving information and role-conditioned personas. It evaluates fine-grained recall, memory awareness, and profile understanding, exposing structural limitations in current systems.

  • Motivation: Existing benchmarks underrepresent real interactions by focusing on dyadic sessions, irrelevant distractors, shallow personas, and token-level recall.These settings miss attribution, cross-topic interference, evolving plans, and role-shaped communication.
  • Benchmark Design: EverMemBench models interconnected group chats with coherent cross-topic interleaving, revisited decisions, diverse role-conditioned personas, and dynamic knowledge updates.The benchmark is designed to resemble long-horizon collaborative deployment rather than isolated topic sessions.
  • Evaluation Dimensions: EverMemBench evaluates fine-grained recall, memory awareness, and profile understanding as complementary capabilities for collaborative assistants.The dimensions cover precise entity retrieval, applying stored knowledge to unseen situations, and consistency with preferences, expertise, and roles.
  • Contributions: The benchmark targets multi-party, multi-group dialogues spanning one million tokens per project, with coherent cross-topic interaction, role-conditioned personas, and dynamic updates.These design choices extend evaluation beyond token-level recall toward relevance recognition and persona consistency.
  • Contributions: EverMemBench is positioned as a benchmark for developing LLMs that reason over time, roles, and collaborative interaction structure.Its stated purpose is to make realistic memory limitations measurable.

2 Related Work

Prior conversational-memory benchmarks and memory-augmented systems are typically evaluated in dyadic or single-user settings. EverMemBench instead stresses multi-party collaboration, interdependent tasks, distributed information, evolving knowledge, and persona interaction.

  • Prior Benchmarks: Long-context memory benchmarks cover capabilities such as multi-session interaction, question answering, event summarization, information extraction, and temporal reasoning.These benchmarks evaluate long-horizon memory from multiple perspectives but retain simplified interaction assumptions.
  • Prior Benchmarks: Existing benchmarks generally assume dyadic or single-user interactions and underrepresent interdependence across roles, groups, and explicit attribution reasoning.This limits their ability to test collaborative memory structure.
  • Memory-Augmented Systems: Memory-augmented systems persist, structure, retrieve, and update conversational information through architectures including memory layers and temporal knowledge graphs.Examples include scalable persistent memory layers and bi-temporal graphs tracking event and ingestion time.
  • Evaluation Gap: Evaluations of these systems under-stress multi-party attribution, cross-group dependency, temporal revision, and persona consistency under shifting social context.Simplified settings make it difficult to distinguish mechanisms that support realistic collaboration from those that succeed only under constrained conditions.
  • EverMemBench: EverMemBench provides a diagnostic collaborative evaluation environment that distributes information across speakers, groups, and time.Its task overview includes nine tasks with illustrative mini-cases and evidence chains.

3 EverMemBench

EverMemBench is a diagnostic benchmark for long-horizon memory in collaborative, multi-party interactions, evaluating recall, memory awareness, and profile understanding under deployment-like conditions. It constructs controlled but realistic dialogues and evidence-grounded QA items through structured generation, validation, and quality control.

  • Task formulation: EverMemBench evaluates long-horizon memory across three task families: fine-grained recall, memory awareness, and profile understanding.The benchmark includes subtasks for scattered-fact retrieval and chaining, constraint generalization, proactive conflict detection, updates, and role-conditioned user modeling.
  • Task formulation: Fine-grained recall tests single-hop grounding, multi-hop cross-group chaining, and temporal event-boundary identification.These subtasks target practical questions such as the final approved budget or actual assignee after revisions.
  • Task formulation: Memory awareness tests constraint generalization, proactive conflict detection, and update reasoning over rule versions and precedence.The scenarios require assistants to apply and revise stored information in novel situations.
  • Task formulation: Profile understanding evaluates whether distributed interaction signals support stable inferences about communication style, expertise, and role focus.These dimensions are intended to guide tone, suggested actions, and assumptions beyond single-shot observations.
  • Task formulation: The benchmark uses a streaming multi-group protocol in which five projects run independently while systems ingest chronologically ordered daily message batches and maintain memory states.Evaluation combines multiple-choice diagnosis with LLM-judged open-ended queries, and annotated evidence spans support oracle-versus-retrieval analysis.
  • Data construction: Data construction combines organizational blueprints, persona-conditioned day-by-day dialogue generation, hierarchical summarization, validation, and three-phase QA quality control.The blueprint enforces decision rights, timelines, and dependencies; generation uses internal summaries; validation checks logic, profiles, and progress; QA filtering removes leaks and enforces evidence sufficiency and uniqueness.

4 Empirical Results

EverMemBench evaluates long-context LLMs and memory-augmented systems by separating context access, retrieval, and reasoning over fragmented evidence. Results show structural failures in multi-party attribution, temporal version reasoning, retrieval of inferentially relevant evidence, and profile understanding.

  • Fine-Grained Recall: 26.51% accuracy marks Gemini-3-Flash’s multi-hop performance, while the best memory-augmented system reaches 18.88%.Single-hop recall remains high, but relevant facts distributed across speakers, groups, and days must be integrated across exchanges.
  • Fine-Grained Recall: 97.99% and 88.37% oracle accuracy for GPT-4.1-mini and Gemini-3-Flash shows that removing retrieval and attribution bottlenecks restores much of their reasoning performance.LLaMA-4 reaches only 37.35% under oracle evidence, indicating a separate conservative-reasoning failure when evidence is fragmented.
  • Fine-Grained Recall: 54.5% accuracy for single-group questions falls to 33.6% for two groups and 19.7% for three groups, a 64% relative decline.The degradation is consistent across models and memory configurations, identifying cross-group attribution as the dominant challenge rather than context length.
  • Fine-Grained Recall: Temporal accuracy remains low across systems, ranging from 2.67–21.00% for memory systems, and the best oracle model reaches only 60%.Correct answers require version semantics for revised, superseded, and finalized decisions, not timestamps alone.
  • Memory Awareness: 87–99% oracle accuracy on Memory Awareness shows that reasoning is sufficient when relevant evidence is available, making retrieval the primary bottleneck.Similarity-based retrieval can miss inferentially relevant information, while weaker answer models also struggle with sparse indirect cues.
  • Profile Understanding: Style is the hardest Profile Understanding subtask, with memory systems at 11–58% and Gemini-3-Flash at 67.05%.Communication style emerges across many interactions and cannot be recovered reliably from isolated snippets, unlike partially inferable skill and role information.
  • Overall Takeaway: Scaling context or retrieval alone is insufficient because attribution, temporal revision, and inferential relevance create structural challenges for current memory paradigms.EverMemBench is intended to diagnose why systems fail rather than merely rank them.

5 Conclusion

EverMemBench evaluates conversational memory in sustained, multi-party collaboration where information is distributed, revised, and shaped by roles. The results identify structural failures and motivate memory architectures with explicit temporal, episodic, cross-group, and reasoning-aware representations.

  • Benchmark Scope: EverMemBench models sustained multi-party collaboration with distributed information, temporal revisions, and role-shaped social context.Its evidence-grounded QA items target attribution, temporal revision, and inferential relevance beyond context length.
  • Future Direction: Future memory architectures should explicitly represent versioned state, episodic boundaries, and cross-group dependencies while integrating retrieval with reasoning.EverMemBench provides a diagnostic foundation for studying these requirements across fine-grained recall, memory awareness, and profile understanding.

A Data Statistics

The benchmark uses a simulated enterprise corpus with 170 role-conditioned employee profiles and five year-long project topics. Its 2,400 QA pairs cover fine-grained recall, memory awareness, and profile understanding across dense multi-group dialogues.

  • User Profiles: 170 simulated employee profiles combine demographics, skill portfolios, eight-dimensional communication styles, and Big-Five personality traits.Profiles span organizational roles and are designed to support implicit user modeling.
  • Demographics: Employee ages range from 23 to 48, and the organization uses five ranks from L1 CEO to L5 Junior.The rank distribution is concentrated in L4–L5, reflecting the benchmark’s simulated corporate hierarchy.
  • Skills: Skills come from 104 competencies, averaging 4.1 skills per member with assignments conditioned on organizational rank.Executives receive broader strategic portfolios, while staff receive more role-specific technical or business competencies.
  • Communication and Personality: Each persona has an eight-dimensional communication profile, with role expectations shaping formality, directness, jargon, and humor.The benchmark also assigns Big-Five traits, with management showing higher Openness and Extraversion than staff.
  • Dialogue Corpus: The corpus contains 51,023 messages and over 2.1 million words across five topics simulating approximately one year each.Each topic uses three concurrent project groups, 32–46 participants, and 834–966 distinct themes.
  • QA Benchmark: 2,400 QA pairs are organized into three evaluation dimensions and nine subtasks.Fine-grained Recall contains 762 pairs, including 213 Single-hop Retrieval, 249 Multi-hop Trajectory, and 300 Temporal Duration questions.
  • QA Design: Each QA pair averages 6.7 evidence references to dialogue messages, enabling evaluation of retrieval precision and comprehension synthesis.Fine-grained Recall uses open-ended questions, while Memory Awareness and Profile Understanding use other evaluation formats described in the benchmark design.

B Data Construction Details

EverMemBench is constructed through validated stages that turn enterprise project blueprints into multi-party dialogues and evidence-grounded QA items. The pipeline combines role-structured teams, cross-group dependencies, year-long scheduling, and human review to keep the benchmark coherent and non-trivial.

  • Blueprint generation: Five enterprise project topics are decomposed into 15 independently executable sub-projects with self-contained group chats and cross-group information flow.The topics span Technology, Operations, Marketing, Financial Services, and Governance.
  • Blueprint generation: Teams are assembled from 170 employees across seven departments using skill alignment, communication complementarity, and hierarchical coverage.Each team includes all three ranks, and communication styles are adapted to team context while preserving original profiles.
  • Blueprint validation: Subtasks are decomposed into concrete deliverables across six sequential development phases, then assigned to members by proficiency, workload balance, and role appropriateness.Assignments include rationales, while global validation checks ownership, temporal consistency, skill coverage, and load balance.
  • Blueprint validation: Cross-project dependencies formalize data contracts, shared infrastructure, and technical decisions so generated conversations reflect collaborative coordination.The resulting blueprints serve as executable specifications for dependencies and role-appropriate communication.
  • Dialogue and QA generation: Dialogues span a simulated 365-day year on workdays, with daily synthesis conditioned on task schedules, summaries, and persona information.The benchmark then synthesizes 2,400 question-answer-evidence triples through specialized pipelines using the missing-key rule.
  • Quality validation: Human reviewers assess question reasonability, answer rigor, distractor deceptiveness, and evidence sufficiency before revising or rejecting failed items.A zero-context GPT-4.1-Mini check found all nine subtasks at or below chance, with Multi-Hop retrieval as low as 2.01%.

C Evaluation Details

Evaluation compares memory-augmented systems using retrieved memories with a full-context baseline using the complete transcript. Answers are assessed through multiple-choice scoring, LLM judging for open-ended responses, and human reliability checks.

  • Answering setup: Memory-augmented systems receive retrieved memories, whereas the full-context baseline receives the complete dialogue transcript directly.Separate prompts are used for multiple-choice and open-ended questions.
  • Answer evaluation: Open-ended answers are evaluated by an LLM judge that compares each generated answer with the question and gold answer.The judge returns a binary CORRECT/WRONG label with a one-sentence rationale.
  • Answer evaluation: Human reliability assessment uses five independent evaluators for each of 30 randomly selected open question-answer pairs.Annotators judged correctness against the question and reference answer.

D Additional Experimental Results

Additional experiments examine performance across project domains and cross-group reasoning difficulty, while reporting the prompts and judging protocol used for answer evaluation. Results show stable domain rankings and sharp degradation as questions span more groups.

  • Domain analysis: System rankings remain largely stable across five project domains: MemOS leads three topics and ranks second on the other two, while MemoBase ranks last throughout.Average accuracy ranges from 34.8% on T2 to 39.2% on T5.
  • Grouping analysis: Accuracy drops sharply from single-group to three-group questions across all three answer models.The pattern persists regardless of backbone capability, indicating a consistent cross-group reasoning challenge.
  • Evaluation prompts: Memory-augmented prompts use retrieved memories, while full-context prompts provide the complete transcript for both open-ended and multiple-choice answering.The evaluation protocol uses separate prompts for the two question formats.
  • Evaluation prompts: The LLM-as-a-Judge protocol receives the question, gold answer, and generated answer and returns a binary correctness label with a rationale.This protocol is used for open-ended answer evaluation.

E Case Examples

Case examples present questions, correct and erroneous answers, and supporting evidence to illustrate typical memory-system errors. The examples include single-hop retrieval and multi-hop reasoning over people, tasks, groups, and dates.

  • Case format: Representative cases pair each question with its correct answer, a typical erroneous answer, and the corresponding evidence.The cases are organized by sub-task and accompanied by qualitative error analysis.
  • Fine-grained recall: A single-hop example asks for the delivery-page link provided after a named participant completed a UI design task.The case tests retrieval of a specific entity from a project dialogue.
  • Fine-grained recall: A multi-hop example asks for the time between one colleague’s task completion and the start of their next independent task.Answering requires identifying the person through a task constraint and computing the temporal gap.

E.1 Fine-grained Recall Examples

Fine-grained Recall tests whether systems can retrieve and reason over precise facts scattered across dense, multi-party conversations. The examples expose failures from distractors, compositional attribution, temporal ambiguity, and date arithmetic.

  • Evaluation scope: The benchmark analyzes representative sub-tasks to reveal how memory systems fail when precise evidence is distributed across speakers and groups.Fine-grained Recall covers Single-hop Retrieval, Multi-hop Trajectory, and Temporal Duration.
  • Single-hop Retrieval: All tested systems returned an irrelevant Figma link instead of the requested Confluence delivery link.The example highlights message-length bias and retrieval insensitivity to task lifecycle stage.
  • Multi-hop Trajectory: Multi-hop Trajectory requires identifying a person from intersecting constraints, locating the true completion event, tracing an 81-day gap, and computing elapsed days.Premature completion signals and unrelated activity create additional retrieval and reasoning demands.
  • Temporal Duration: Temporal Duration requires extracting task boundaries and computing either calendar-day or working-day spans from interleaved dialogue.The working-day case excludes weekends, so Apr 22 to Apr 28 spans 7 calendar days but 5 working days.
  • Temporal Duration: Keyword collisions cause systems to retrieve nearby “archived in Confluence” events instead of the correct Apr 10 completion event.The calendar-day case separates retrieval of the correct anchors from subsequent temporal reasoning.

E.2 Memory Awareness Examples

Memory Awareness evaluates whether systems can apply stored information to novel situations rather than merely retrieve it. The examples test latent role constraints, policy compliance, and temporally ordered rule updates.

  • Evaluation scope: Memory Awareness distinguishes systems that merely store information from those that generalize it to unseen scenarios and act on it.Its sub-tasks evaluate application of stored knowledge to novel situations.
  • Constraint: Constraint reasoning requires inferring responsibility for a novel schema change from implicit organizational role relationships.The workOrderNumber request never appeared in the original conversation and is under-specified without contradicting known rules.
  • Proactivity: Proactivity requires detecting that an urgent Zustand proposal conflicts with the project’s established technology policy.The request is technically compelling but non-compliant, testing whether systems resist persuasive framing.
  • Update: Update reasoning requires overriding an earlier GitFlow emergency-fix rule with a later CORE_ALGO P0 directive and composing its operational steps.The system must chain the base rule, override mapping, and override definition into one action sequence.

E.3 Profile Understanding Examples

Profile Understanding tests whether systems infer implicit persona traits from long-term dialogue and use them to shape responses. The examples isolate communication style, role, and skill constraints in recommendations and drafted replies.

  • Evaluation scope: Profile Understanding concerns stable, implicit persona traits expressed across long-term dialogue, including communication style, skills, and professional role.These traits are rarely stated explicitly and must be distilled from recurring conversational patterns.
  • Style: Style evaluation separates factual recall from persona replication by making tone the only difference between otherwise equivalent reply options.Ruiqing Jiang’s characteristic register includes frequent emojis and casual phrasing.
  • Skill: Skill evaluation requires constraining Jie Gu’s technical recommendation by his Java expertise and the team’s Java/Spring Boot stack.The profile lists strong Java and medium Spring Boot skills, with no Python skill listed.
  • Persona conditioning: Profile-aware responses must reflect whose perspective should answer a technical question, not only which topical information is relevant.Similarity-based retrieval can surface feature comparisons while missing persona-level constraints.
  • Role: Role evaluation requires framing Yutong Song’s improvement suggestions from a developer’s perspective rather than a testing-process perspective.Systems often retrieved the incident correctly but selected the role-inappropriate recommendation.
Loading 2602.01313v3…