PaperlayerView exact revision on arXiv

Source-linked AI summary

GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

Jingbo Yang, Kwei-Herng Lai, Xiaowen Wang, Shiyu Chang, Yaar Harari, Evgeniy Gabrilovich

arXiv:2605.14498v2cs.CL

TL;DR

Existing memory systems and benchmarks largely focus on dyadic conversations, leaving group dynamics, speaker-grounded beliefs, and audience-adapted language unmeasured. GroupMemBench constructs structured multi-party conversations and asker-bound adversarial queries. Leading systems achieve low accuracy, and the paper concludes that current ingestion pipelines erase features group memory depends on.

  • Problem

    Most agent memory systems and evaluations target dyadic, single-user settings, leaving group dynamics, speaker-grounded beliefs, and audience-adapted language unmeasured.

    Source: §1 · pp. 1–2

  • Method

    GroupMemBench combines graph-grounded multi-party conversation synthesis with user-bound queries designed to test six dimensions of memory.

    Source: §1 · p. 3, §3.2 · p. 5

  • Results

    46.0% average accuracy for the best-performing system, including 27.1% on knowledge update and 37.7% on term ambiguity; BM25 matches or exceeds most agent memory systems.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Current ingestion pipelines erase threaded-conversation structure and lexical shifts that group memory depends on.

    Source: §5 · p. 10

    The benchmark covers English-language, text-only workplace conversations across four domains; multilingual and multimodal settings are left to future work.

    Source: K Limitations · p. 23

Abstract

from arXiv · show

Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems that extract, retrieve, and apply information across long-running conversations. However, both existing memory systems and benchmarks are built around the dyadic, single-user setup, even though real deployments routinely span groups and channels with multiple users interacting with the agent and with each other. This mismatch leaves three properties of group memory unmeasured: (i) group dynamics that go beyond concatenated one-on-one chats, (ii) speaker-grounded belief tracking, where the per-user memory modeling is needed, and (iii) audience-adapted language, where Theory-of-Mind shifts produce role-specific vocabulary. We introduce GroupMemBench, a benchmark that exposes all three. A graph-grounded synthesis pipeline produces multi-party conversations with controllable reply structure and conditions each message on per-user personas and target audiences. An adversarial query pipeline then binds every question to a specific asker across six categories, spanning multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention, and iteratively searches challenging, realistic queries that reflect comprehensive memory capability. Benchmarking leading memory systems exposes a sharp collapse: the strongest one reaches only 46.0% average accuracy, with knowledge update at 27.1% and term ambiguity at 37.7%, while a simple BM25 baseline matches or exceeds most agent memory systems. This indicates current memory ingestion erases the structural and lexical features group memory depends on, leaving multi-user memory far from solved.

1 Introduction

Existing memory systems and evaluations largely target dyadic conversations despite agents being deployed in group settings. GroupMemBench tests group dynamics, speaker-grounded beliefs, and audience-adapted language, and finds low accuracy from leading memory systems.

1 Introduction

  • Current memory systems and evaluations largely target dyadic, single-user settings, leaving multi-party memory dynamics unmeasured.Group deployments include channels, threads, and project spaces where multiple users interact with the agent and one another.Source: §1 · p. 1
  • Group memory requires tracking collective discussion flow, speaker-specific beliefs, and role-dependent vocabulary shaped by audience assumptions.For example, “token” may refer to text units in NLP or authentication credentials in system design.Source: §1 · p. 2
  • GroupMemBench uses controllable reply structures, asker-bound queries, and persona- and audience-conditioned messages to test these challenges.Its query categories include multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention.Source: §1 · pp. 2–3
  • 46.0% average accuracy for the best-performing system, with knowledge update at 27.1% and term ambiguity at 37.7%.Abstention remains above 80%.Source: §1 · pp. 2–3

2 Related Work

Prior agent-memory systems and long-term memory benchmarks largely assume dyadic interactions, with queries issued by a single user. GroupMemBench targets the missing speaker-, audience-, and thread-sensitive demands of multi-party memory.

2 Related Work

  • Agent-memory designs commonly isolate users by namespace and overwrite older conflicting statements, assumptions that GroupMemBench probes with speaker-attributed facts and coexisting preferences.The passage describes MemGPT, Mem0, A-Mem, HippoRAG, and Hindsight as representative designs.Source: §2 · p. 3
  • Long-term memory benchmarks expand context length and reasoning tasks but retain single-author dialogues and queries issued by “the user.”Prior work therefore does not require speaker-conditioned retrieval, coexisting preferences, or role-dependent vocabulary.Source: §2 · p. 3
  • EverMemBench is a nearby multi-party effort, but its homogeneous speakers lack persona and Theory-of-Mind modeling; GroupMemBench pairs graph-grounded synthesis with speaker-dependent adversarial queries.This leaves user-implicit reasoning and addressee-dependent vocabulary untested in the earlier benchmark.Source: §2 · p. 4

3 GroupMemBench

GroupMemBench formalizes memory-grounded question answering for group conversations, where answers depend on the querying user. It synthesizes structured, audience-conditioned discussions and adversarial questions, then assesses synthesis quality across six dimensions.

3.1 Problem Formulation

  • The task evaluates memory-grounded question answering with an explicit querying-user input, since the correct answer may vary with that user’s role and history.The agent’s answer is appended to the shared conversation and updates the memory state.Source: §3.1 · p. 4

3.2 Synthesizing Multi-Party Conversations

  • A growing graph of users, domains, topics, phases, and messages supports three weighted traversal templates for selecting authors, recipients, and conversational situations.The templates model channel posts, threaded replies, and cross-project bridges.Source: §3.2 · p. 5
  • Generation combines recent local history with phase summaries of decided, open, and pending issues, then conditions messages on persona, topic, phase, stage, and scenario.Threaded replies reference parent messages; realism perturbations include noise, disagreement, stylistic imperfections, and decision changes.Source: §3.2 · pp. 5–6
  • Graph-guided synthesis closely tracks the real-world upper bound on most G-Eval dimensions and substantially outperforms the single-prompt baseline.Figure 3 reports scores across six dimensions, averaged over 10 seeds with shaded bands showing ±1 standard deviation.Source: Figure 3 · p. 6, §3.2 · p. 6

3.3.1 Shared Generation Pipeline

  • The shared query pipeline selects answerable anchors, drafts type-specific questions, and accepts queries only after a retrieval baseline fails in a Solve–Judge–Refine loop.Accepted queries retain the question, gold answer, querying-user ID, and evidence trace.Source: §3.3 · p. 6, §3.3.1 · p. 6

3.3.2 Dimensions and Type-Specific Strategies

  • Six query types test multi-hop reasoning, knowledge updates, speaker-conditioned term ambiguity, first-person user references, temporal reasoning, and abstention.Knowledge-update questions probe changed decisions, while term-ambiguity questions pair the asker’s terminology with evidence phrased for another role.Source: §3.3.2 · pp. 6–7

4 Experiments

Across the experiments, systems struggle with group structure, speaker-grounded memory, and lexical shifts, while performance and efficiency trade-offs vary substantially. Retrieval failures account for most errors, though memory representations also affect reasoning after retrieval.

4.3 Performance-Efficiency Trade-off

  • HippoRAG is on the cost–accuracy Pareto front in all four domains; Hindsight achieves the highest accuracy at 2–5× HippoRAG’s cost, while storage does not track accuracy.Figure 4 plots ingestion cost against per-domain accuracy, with bubble area encoding storage.Source: Figure 4 · p. 8, §4.3 · pp. 8–9

4.2 Main Results

  • Current systems flatten group structure, fail to condition memory on speaker identity, and struggle with lexical shifts; Temporal and Abstention perform best because they depend least on multi-party structure.Source: §4.2 · p. 8
  • No system exceeds 50% on Multi-Hop, and BM25 beats four of five agent memory systems on average, consistent with group structure being flattened during ingestion.Source: §4.2 · p. 8
  • On Knowledge Update, no system passes 28%; Mem0 reaches 4.67 because its extractor appends new statements alongside obsolete ones without speaker-conditioned consolidation.Source: §4.2 · p. 8
  • Term Ambiguity stays below 38% across all baselines and remains uniformly low after gold retrieval, making representation rather than retrieval the remaining wall.Source: §4.2 · p. 8

4.4 Failure Analysis

  • Across baselines, retrieval failures account for 41–79% of questions, versus 7–24% for reasoning failures; most systems’ accuracy scales near-linearly with recall.Figure 5 decomposes failures and plots retrieval recall against answer accuracy.Source: Figure 5 · p. 9, §4.4 · p. 9

4.3 Performance-Efficiency Trade-off

  • BM25 costs effectively $0 and reaches 43.2% cross-domain accuracy, strictly Pareto-dominating four of the five agent memory systems.Source: §4.3 · p. 9

4.4 Failure Analysis

  • LLM raw message text attains the highest mean conditional accuracy in both domains, matching or exceeding every memory-augmented baseline.Figure 6 reports P(correct | gold retrieved) by baseline and question type.Source: Figure 6 · p. 9, §4.4 · p. 9

5 Conclusion

GroupMemBench exposes weak group-memory performance, with BM25 matching or exceeding most systems and current ingestion pipelines erasing important structural and lexical features. Its synthesis graph represents projects, phases, participants, and persona-conditioned voices.

5 Conclusion

  • BM25 matches or exceeds most leading memory systems, and current ingestion pipelines erase threaded structure and lexical shifts that group memory depends on.The conclusion calls for more generalized memory systems for both dyadic and multi-party settings.Source: §5 · p. 10

A Graph Schema

  • The synthesis graph models projects, topics, time-bounded phases, participants, and generated messages, with each message recording its author, timestamp, and path type.Users have role, tone, style, and expertise personas; messages follow post, reply, or cross-project-reply paths.Source: A Graph Schema · p. 12
  • Active-phase membership restricts candidate message authors, while shared project-category tags connect sibling projects for cross-project walks.The related_to_project relation is the only edge traversed by the cross-project handler.Source: A Graph Schema · p. 12
  • Distinct tone-and-style pairs and injected behavioral descriptors give users recognizable individual voices during message generation.Expertise and role are sampled independently and uniformly.Source: A Graph Schema · p. 12

B Path Sampling and Graph Walks

The synthesis process samples channel posts, threaded replies, and cross-project replies through graph paths, then prompts persona-conditioned messages and maintains phase-level memory artefacts. Synthetic conversations are evaluated with six judge-scored quality metrics.

B Path Sampling and Graph Walks

  • At each generation step, a path type is sampled from the (0.20, 0.60, 0.20) mixture; phases with no messages are forced to post, while recent all-open-question posts without engagement (streak ≥3) force reply.Source: B Path Sampling and Graph Walks · p. 12
  • Channel posts select phases by post-need score, while threaded replies sample phases by heat and select a parent message from the phase history.The post type depends on message volume, progressing from kickoff or milestone messages to the full set of post types.Source: B Path Sampling and Graph Walks · p. 13
  • Reply-parent weights combine preferential attachment with saturation, recency decay, and a post-over-reply bias; responders are sampled from active phase members with recent speakers excluded when possible.The responder is drawn uniformly from eligible members after excluding the parent author and, when possible, the last three speakers.Source: B Path Sampling and Graph Walks · p. 13
  • Cross-project replies traverse one sibling-project link toward a target phase, favoring a user with the source author’s role and rejecting temporally earlier phases.The target phase is weighted by its active-thread score.Source: B Path Sampling and Graph Walks · pp. 13–14

C Prompt Templates

  • Persona-conditioned prompts set each speaker’s voice, while channel-post, threaded-reply, and cross-project templates specify scenario context and tasks.Separate phase-memory prompts generate six sub-issues per phase and periodically update a three-section ledger that moves answered questions from open to decided.Source: C Prompt Templates · pp. 14–15

D Hyperparameters and Realism Knobs

E Evaluation Rubrics for Synthetic Conversation

  • Six LLM-as-Judge metrics score synthetic conversations from 1 to 5: naturalness, coherence, diversity, contextual relevance, momentum, and engagingness.Their arithmetic mean is the overall average quality score.Source: E Evaluation Rubrics for Synthetic… · pp. 15–16

F Dataset Statistics

The synthesized corpus balances message counts and path-type proportions across four domains while varying participant and role counts to span different levels of organizational complexity.

F Dataset Statistics

  • 30,000 messages per domain balance corpus size across the four domains, preventing size imbalance from explaining cross-domain accuracy or cost differences.Table 6 reports per-domain statistics for the synthesized conversations.Source: F Dataset Statistics · p. 16, Table 6 · p. 17
  • Posts and replies remain near a 1:3 ratio, with about 10% of messages crossing project boundaries, consistent across domains.The path-sampling mixture is (0.20, 0.60, 0.20) for posts, replies, and cross-project replies.Source: F Dataset Statistics · p. 16
  • Participant counts range from 6 in Healthcare to 18 in Technology, while role counts range from 4 to 6 to vary organizational complexity.The corpus spans a leaner Healthcare setting and a richer Technology setting.Source: F Dataset Statistics · p. 16

G Per-Domain Results

Per-domain baseline accuracies are reported for Technology, Finance, Healthcare, and Manufacturing, alongside cross-domain micro-averages over the combined filtered evaluation set.

G Per-Domain Results

  • Tables 7–10 report per-domain baseline accuracy, while Table 2 summarizes cross-domain micro-averages over the union of the filtered evaluation set.The four domains are Technology, Finance, Healthcare, and Manufacturing.Source: G Per-Domain Results · p. 17

H Per-Domain Ingestion Cost and Storage

Ingestion costs count successful LLM and embedding calls, while storage is measured from each baseline’s on-disk store; shared database volumes are allocated across domains.

H Per-Domain Ingestion Cost and Storage

  • Ingestion costs use per-call token counts and OpenAI list prices, summed across successful LLM and embedding calls while excluding query-time calls.The listed prices are $0.15 / $0.60 per 1M input/output tokens for gpt-4o-mini and $0.13 per 1M tokens for text-embedding-3-large.Source: H Per-Domain Ingestion Cost and Storage · p. 17
  • Storage is measured directly with du -sb on each baseline’s per-domain store directory.Source: H Per-Domain Ingestion Cost and Storage · p. 17
  • HINDSIGHT and MEMGPT share Postgres volumes, apportioned by domain using each system’s reported bookkeeping counter.HINDSIGHT uses node and link counts; MEMGPT uses archival passage counts.Source: H Per-Domain Ingestion Cost and Storage · p. 17

I LLM Judge Configuration and Reliability

Accuracy is scored by a single LLM judge using a strict-equivalence prompt and deterministic verdict parsing. A 100-item manual check found 99% agreement, and the authors report that baseline rankings are unaffected.

I LLM Judge Configuration and Reliability

  • The evaluation uses one LLM judge per question, gold answer, and predicted answer, without self-consistency or multi-sample voting.The judge is GPT-5 served via Azure OpenAI, with max_tokens = 2048 and temperature = 1.0.Source: I LLM Judge Configuration and… · p. 18
  • The judge is instructed to accept paraphrases with the same meaning as the gold answer and emit a structured verdict.Source: I LLM Judge Configuration and… · p. 18
  • Verdict parsing checks the final structured suffix, prioritizing incorrect, wrong, or not correct over correct; unclear outputs are excluded from the denominator.Negative triggers are checked first because “not correct” contains the positive trigger.Source: I LLM Judge Configuration and… · p. 19
  • The judge agreed with manual review on 99 of 100 sampled items; the lone disagreement was a false negative for a correct answer with extra material.Source: I LLM Judge Configuration and… · p. 19
  • The sample contained no false positives, and the authors say the strict-side bias makes reported accuracy mild lower bounds without changing cross-baseline rankings.Source: I LLM Judge Configuration and… · p. 20

J Worked example: information loss during memory ingestion

This worked example traces a Finance user-implicit question across eight baselines and shows how ingestion can preserve or lose the speaker, request, and counterparties needed for an answer. The failures fall into topic-stub compression, wrong-speaker shadowing, and multi-speaker over-broadening.

J Worked example: information loss during memory ingestion

Top retrieved memory:

  • Three pipelines answer correctly: one leaves information unrevised, dense retrieval over raw messages places the answer message at rank 7 of the top 10, and HINDSIGHT compression retains the speaker-linked request and counterparties.Source: J Worked example: information loss… · pp. 20–21, Top retrieved memory · p. 21
  • The five incorrect answers illustrate three failure modes: topic-stub compression in Mem0, wrong-speaker shadowing in A-Mem, and multi-speaker over-broadening in HippoRAG, MemGPT, and GraphRAG.Source: Top retrieved memory · p. 21
  • Mem0’s topic stubs lose the asker, request, and counterparties, while A-Mem retrieves similar posts from User_7 instead of User_13’s request.Source: Top retrieved memory · pp. 21–22
  • HippoRAG, MemGPT, and GraphRAG broaden the answer by combining other speakers’ counterparty lists or replacing individual messages with a domain-level summary.HippoRAG and MemGPT include extras; GraphRAG’s community report folds the asker and specific request into a topical narrative.Source: Top retrieved memory · pp. 22–23
  • The example’s central finding is that answer correctness is set at ingestion: preserving a retrievable unit that binds the asker’s request to its counterparties supports the answer.The five failures break that co-location through topic compression, wrong-speaker retrieval, or mixing speakers’ counterparty lists.Source: Top retrieved memory · p. 23

K Limitations

GroupMemBench is currently limited to English-language, text-only workplace conversations in four domains. The paper also notes potential collaboration benefits and risks from stronger speaker-conditioned memory.

K Limitations

  • The benchmark currently covers English-language, text-only workplace conversations across Technology, Healthcare, Manufacturing, and Finance.Multilingual and multimodal settings are identified as future extensions.Source: K Limitations · p. 23

L Broader Impacts

  • Speaker-grounded memory could reduce miscommunication and cognitive load in collaboration tools, but could also enable workplace surveillance or targeted persuasion if irresponsibly deployed.The artifact uses synthetic conversations with fictional personas rather than real chat logs, and discloses persona vocabularies and prompt templates for audit.Source: L Broader Impacts · p. 23
Loading 2605.14498v2…