PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations

Wenxu Jia, Xize Cheng, Zihan Zhang, Dongjie Fu, Linjun Li, Wenshi Chen, Yangyang Wu, Tao Jin

arXiv:2609.32522v1cs.AIcs.IRcs.SDeess.AS

TL;DR

Long-term memory for multi-party spoken conversations remains underexplored despite the need to track participants and who speaks to whom across sessions. VoxPolyMem combines speaker-aware hierarchical memory with adaptive retrieval and EG-GRPO, achieving an overall score of 85.0 on VoxPolyBench, 23.6 points above the strongest external baseline.

  • Problem

    Long-term memory for multi-party spoken conversations remains underexplored, despite the need to preserve participant identities and interaction roles across sessions.

    Source: §1 · pp. 1–2

  • Method

    VoxPolyMem combines incremental speaker identification, interaction-aware hierarchical memory, and adaptive retrieval trained with round-wise rewards for newly acquired supporting evidence.

    Source: §1 · p. 2

  • Results

    VoxPolyMem scores 85.0 overall on VoxPolyBench, 23.6 points above the strongest external baseline, and leads all four task subsets.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    Experiments support the effectiveness of VoxPolyMem’s memory structure and agentic retrieval on multi-session spoken interactions and public image-text benchmarks.

    Source: §6 · pp. 10–11

    Because VoxPolyBench uses generated dialogues and synthesized speech, it does not establish robustness across real speakers, accents, or recording conditions.

    Source: Ethics Statement · p. 11

Abstract

from arXiv · show

Long-term memory enables agents to accumulate information and reason across sessions, yet existing research primarily focuses on dyadic text or image-text conversations, leaving long-term memory for multi-party spoken conversations underexplored. This setting requires preserving conversational content, identifying participants across sessions, and retaining who speaks to whom. To this end, we propose VoxPolyMem, an interaction-aware multimodal memory framework combining incremental speaker identification with a memory hierarchy comprising interaction memory, fact memory, and participant profiles. We formulate retrieval as sequential decision-making, where an agent rewrites queries and selects retrieval tools and memory layers based on accumulated evidence to address information gaps. We further introduce Evidence-Gain GRPO (EG-GRPO), which uses round-wise credit assignment to encourage complementary evidence acquisition. We also construct VoxPolyBench to evaluate memory evolution, personalized answering, memory retrieval and reasoning, and interaction reasoning and attribution in multi-party spoken conversations. VoxPolyMem achieves an overall score of 85.0 on VoxPolyBench, surpassing the strongest evaluated baseline by 23.6 points. On Mem-Gallery and H2HMem-Multi, it scores 89.6 and 74.4, respectively, exceeding the strongest evaluated public memory baselines by over 8 points each. These results highlight its potential for persistent, personalized assistance in multi-party multimodal interactions. Code and datasets are available at https://voxpolymem.github.io/VoxPolyBench/demo/

1 Introduction

Long-term memory for multi-party spoken conversations remains underexplored, although it requires tracking participants and who speaks to whom across sessions. VoxPolyMem combines speaker identification, interaction-aware memory, and adaptive retrieval, and VoxPolyBench evaluates these capabilities.

1 Introduction

  • Long-term memory research largely evaluates dyadic text or image-text conversations, while multi-party spoken settings require recurring participant identification and interaction tracking across sessions.Preserving voice representations and speaker/addressee roles supports extracting factual memories and participant profiles.Source: §1 · pp. 1–2
  • VoxPolyBench evaluates recurring-participant identification, memory evolution, personalized answering, retrieval and reasoning, and interaction attribution in multi-party spoken conversations.The benchmark comprises 1,527 QA pairs across 176 sessions and 18.9 hours of synthesized speech in 18 scenarios.Source: §1 · p. 2
  • VoxPolyMem combines incremental speaker identification, interaction-aware hierarchical memory, and adaptive retrieval trained with round-wise rewards for newly acquired supporting evidence.The framework includes interaction memory, fact memory, and participant profiles; its retrieval agent selects layers and tools and rewrites queries as evidence accumulates.Source: §1 · p. 2
  • Experiments report consistent answer-quality improvements across VoxPolyBench, Mem-Gallery, and H2HMem-Multi, with interaction-aware memory benefits and better evidence recall in fewer retrieval rounds.The introduction also reports that retrieval-policy comparisons show improved evidence recall with fewer retrieval rounds.Source: §1 · p. 2

2 Related Work

Prior work builds long-term memories through consolidation, linking, temporal organization, compression, and multimodal hierarchies, while retrieval methods span routing, reasoning, and learned rewards. The section also contrasts existing benchmarks with joint evaluation of long-term memory and spoken multi-party interaction.

2.1 Long-Term Memory Construction and Retrieval

2.2 Long-Term Memory and Spoken Interaction Benchmarks

  • Existing conversational-memory and spoken-interaction benchmarks do not jointly evaluate acoustic identity recovery and participant-dependent memory; VoxPolyBench covers both with multi-session attribution tasks.Its evaluation includes recurring speaker identification, memory evolution, personalized answering, retrieval and reasoning, and speaker and addressee attribution.Source: §2.2 · p. 3

3 VoxPolyBench

VoxPolyBench evaluates long-term memory across recurring participants and multiple spoken-conversation tasks. Its construction uses planned events to guide dialogue and speech generation, followed by QA construction and validation.

3 VoxPolyBench

  • VoxPolyBench contains 1,527 QA pairs across 176 sessions and 18.9 hours of synthesized speech in 18 scenarios, with histories spanning 8–12 sessions.Its tasks cover memory evolution, personalized answering, retrieval and reasoning, and interaction attribution.Source: §3 · p. 3
  • Scenario plans and within- and cross-session event anchors guide multi-party dialogue and speech generation before QA construction and validation.Human reviewers check anchors before dialogue generation and revise identified errors or implausible event transitions.Source: §3 · p. 3, Figure 1 · p. 4
  • Speech quality checks include ASR mismatch review and targeted resynthesis; Whisper large-v3-turbo evaluation reports a corpus-level WER of 2.14%.QA pairs are checked for event consistency, participant and interaction assignments, question clarity, and evidence sufficiency; unsupported or ambiguous pairs are removed.Source: §3 · p. 4
  • Table 1 compares spoken-dialogue and memory benchmarks by evaluation coverage, distinguishing explicit, partial or adjacent, and absent evaluation.Its scope includes interaction attribution for speakers and addressees and personalization conditioned on the querying participant.Source: Table 1 · p. 4

4 VoxPolyMem

VoxPolyMem links online speaker identification, hierarchical memory construction, and evidence-seeking query answering in one pipeline. Its retrieval policy iteratively gathers evidence from selected memory layers and tools.

4 VoxPolyMem

  • VoxPolyMem links online speaker identification, hierarchical memory construction, and evidence-seeking query answering in one pipeline.The retrieval agent uses profile voice memory to identify spoken-query participants before gathering evidence.Source: §4 · p. 5

4.1 Online Speaker Identification

  • Audio turns match the closest stored voiceprint when similarity reaches τmatch; otherwise, the system creates a new anonymous speaker ID.Non-audio turns leave the speaker memory unchanged; matched embeddings update by EMA only above τupdate > τmatch, and post-session merging reconciles provisional IDs.Source: §4.1 · pp. 5–6

4.2 Hierarchical Interaction-Aware Multimodal Memory

  • Interaction memory stores multimodal messages and participant relations, while fact memory and profiles organize grounded statements and participant attributes.The LLM processes the current turn and up to three preceding turns, retaining links from facts and profiles to source dialogue turns.Source: §4.2 · p. 6

4.3 Trainable Agentic Memory Retrieval

  • Each retrieval action selects a memory layer and tool, rewrites the query, and may apply speaker or addressee filters using accumulated evidence.The state includes the query, asker identity, retained evidence, action history, and remaining budget.Source: §4.3 · p. 7
  • EG-GRPO rewards newly retained annotated supporting source IDs, balancing evidence coverage against memory rank across retrieval rounds.Each candidate retains up to 15 memory records; source IDs already represented in earlier retained contexts on that branch are excluded from newly acquired evidence.Source: §4.3 · p. 7

5 Experiments

VoxPolyMem leads the evaluated memory benchmarks, while component ablations and retrieval-policy comparisons highlight the contributions of interaction-aware memory and iterative evidence acquisition.

5 Experiments

  • 95.0% attribution accuracy on IEMOCAP and 87.9% on AMI are achieved with reference utterance boundaries.Source: §5 · p. 8

5.1 Experimental Setup

  • The held-out evaluation compares VoxPolyMem with ten baselines across VoxPolyBench, Mem-Gallery, and H2HMem-Multi.On VoxPolyBench, all methods use the same ASR transcripts, but baselines receive ground-truth speaker labels; all use GPT-4.1-mini for answering and a final-context budget of 15 records. EG-GRPO training uses about 900 QA instances, with supporting-turn annotations unavailable at inference.Source: §5.1 · p. 8

5.2 Main Results

  • 85.0 on VoxPolyBench exceeds the strongest external baseline by 23.6 points; Mem-Gallery and H2HMem-Multi score 89.6 and 74.4, beating baselines by 11.8 and 8.4 points.Source: §5.2 · p. 8
  • 86.6 versus 68.1 for the strongest external baseline: VoxPolyMem leads all three benchmarks in weighted average score.The weighted average weights benchmark scores by their test-question counts.Source: §5.2 · p. 8, Table 2 · p. 9

5.3 Component Ablations

  • Removing fact memory and participant profiles lowers H2HMem-Multi from 72.1 to 63.6; removing interaction edges and relation filtering lowers VoxPolyBench from 84.0 to 78.6.The first change restricts retrieval to original interaction records; the second retains speaker identities.Source: §5.3 · pp. 9–10

5.4 Comparison of Retrieval Policy Optimization Methods

  • On Mem-Gallery, EG-GRPO raises answer scores from 87.1 to 89.6 and recall from 91.5 to 93.6 versus MoT-GRPO, using 1.2–1.3 rounds versus 1.4–1.8.The reported round averages exclude answer generation.Source: Table 4 · p. 10, §5.4 · p. 10

6 Conclusion

The paper presents VoxPolyMem and VoxPolyBench for persistent memory over multi-party spoken interactions, while identifying synthetic-data scope and real-world deployment safeguards. The appendix reports dataset quality-control procedures and notes reproducibility dependence on hosted-model APIs and sampling variability.

6 Conclusion

  • The conclusion presents VoxPolyMem’s incremental speaker identification, interaction-aware hierarchical memory, and adaptive retrieval, and reports that experiments support the memory structure and agentic retrieval.The authors report further improvements from policy optimization and identify real-world spoken interactions with greater acoustic and conversational diversity as future evaluation work.Source: §6 · pp. 10–11

AI Use Statement

  • LLMs supported benchmark construction and the memory and evaluation pipelines, and generative AI tools assisted code development.The authors take responsibility for the final manuscript and research artifacts.Source: AI Use Statement · p. 11

Ethics Statement

  • VoxPolyBench uses generated dialogues and synthesized speech, so its controlled setting does not establish robustness across real speakers, accents, or recording conditions.The authors call for informed consent, restricted access, limited retention, and mechanisms to correct or delete stored information in real-world deployment.Source: Ethics Statement · p. 11

Reproducibility Statement

A.1 Construction and Quality Control

  • Dataset checks verify event progression, speaker and addressee assignments, question clarity, answer support, and evidence sufficiency; unresolved ambiguous or unsupported pairs are removed.Human reviewers inspect session anchors, and both human reviewers and an LLM check QA pairs.Source: §A.1 · p. 15
  • Speech quality control screens synthesis duration and ASR mismatches, identifier errors, and suspected repetitions, followed by targeted rechecks and manual spot listening for selected samples.Each participant keeps a fixed reference voice across sessions.Source: §A.1 · p. 15
  • 2.14% corpus-level WER measures automatic-transcription agreement with normalized synthesis source texts, weighted by reference word count.It is not a human-verified synthesis error rate.Source: §A.1 · p. 15

A.2 Dataset Statistics

VoxPolyBench organizes evaluation around longitudinal multi-party histories and a range of memory tasks. Its scope combines spoken identity and interaction attribution with memory evaluation properties that existing benchmarks do not jointly cover.

A.2 Dataset Statistics

  • VoxPolyBench contains 18 case-level histories spanning 176 sessions, 9,599 dialogue turns, and 18.9 hours of synthesized speech; each case follows recurring participants over 8–12 sessions.Source: §A.2 · p. 15
  • The 1,527 QA instances cover retrieval and reasoning (443; 29.0%), personalized memory (434; 28.4%), interaction attribution (326; 21.3%), and evolving memory (324; 21.2%).Nine fine-grained types include single- and multi-hop retrieval, memory updates and conflicts, temporal reasoning, speaker and addressee attribution, and personalized recall.Source: §A.2 · p. 16, Figure 3 · p. 16

A.3 Comparison with Existing Benchmarks

  • Existing speech benchmarks primarily lack persistent participant identity, while multimodal memory benchmarks generally do not jointly evaluate spoken identity, source-speaker and addressee attribution, and query-participant conditioning.VoxPolyBench combines these properties with memory evolution, cross-session evidence integration, and gold evidence provenance.Source: §A.3 · pp. 16–17
  • The authors describe VoxPolyBench as the first benchmark for long-term memory reasoning over multi-session, multi-party spoken conversations; Table 1 compares targeted capabilities, not overall benchmark quality.Source: §A.3 · p. 17

B Implementation Details

The implementation reconciles speaker identities across sessions, configures participant-aware retrieval, and trains retrieval decisions with round-wise evidence rewards.

B.1 Acoustic Speaker Tracking

  • Speaker groups are merged iteratively using cross-group voiceprint similarity, mutual-best and cohesion checks, session overlap, and stricter singleton requirements.The highest-scoring eligible pair is merged and candidates are recomputed until no eligible pair remains; original voiceprint embeddings and online EMA entries are preserved.Source: §B.1 · p. 17

B.2 Memory and Retrieval Configuration

  • Identity mappings link fragmented speaker identifiers to participant names and profiles, while filters can constrain retrieval by asker identity.The final context limit of k = 15 applies after reciprocal-rank fusion, deduplication, and context selection, not per tool call.Source: §B.2 · p. 17

B.3 Evidence-Gain GRPO Details

  • Evidence-Gain GRPO scores newly acquired supporting IDs using retained memories, with source-ID mapping used for reward calculation rather than replacing memories with source turns.Shared memory ranks, earliest-rank assignment, and unique source-ID counting define the rank-weighted evidence gain; empty contexts receive zero reward, and training requires supporting-turn annotations.Source: §B.3 · pp. 17–18
  • Training branches track source IDs represented in retained memories and continue until current memories cover all supporting IDs or the round budget runs out.Historical coverage alone does not stop a branch; inference uses evidence sufficiency and the round budget without gold annotations.Source: §B.3 · p. 18
  • The evidence reward trains retrieval actions through a policy objective, while the answer model remains fixed and input or retrieved tokens are excluded from the action loss.Automatically supplied filters are also excluded from the action loss.Source: §B.3 · p. 18

C.1 Data Splits and Baselines

The evaluation uses three benchmarks and compares VoxPolyMem with text-based and multimodal memory baselines.

C.1 Data Splits and Baselines

C.2 Metrics and Aggregation

The section defines weighted benchmark aggregation and evidence recall, then specifies an LLM judge protocol and scoring rubric.

C.2 Metrics and Aggregation

  • Cross-benchmark averages weight benchmark scores by total test-question counts, and aggregates are computed before display rounding.Table 5 gives training and test QA counts; a dash denotes an unavailable measurement.Source: §C.2 · pp. 18–19, Table 5 · p. 19
  • Mem-Gallery tasks are grouped into E&A, R, and KM dimensions, while VoxPolyBench temporal reasoning belongs to EC.Evidence recall is measured after final-context deduplication and truncation, counting each linked source turn ID once.Source: §C.2 · p. 19

C.3 LLM-as-a-Judge Protocol

  • Judges receive the question, reference answer, and system answer, then return a JSON score and short explanation on a five-value scale.The permitted scores are 0, 0.25, 0.5, 0.75, and 1.Source: §C.3 · pp. 19–20
  • The rubric ranges from incorrect or missed answers at 0 to accurate, precise, exact answers at 1, with partial and good responses between.Intermediate scores distinguish tangential or vague responses, incomplete responses, and answers with only minor imperfections.Source: §C.3 · pp. 19–20

D.1 Speaker Identity Evaluation across Three Datasets

The speaker-tracker ablation finds gains from EMA updates and identity reconciliation, while a human audit compares judge and human scores.

D.1 Speaker Identity Evaluation across Three Datasets

  • The tracking evaluation covers 18 VoxPolyBench cases, IEMOCAP Sessions 4–5, and five AMI scenario chains using frozen ECAPA embeddings and oracle utterance boundaries.Reference speaker labels and speaker counts are used only for scoring; inference receives neither a participant roster nor a target speaker count.Source: §D.1 · pp. 20–21
  • Identity reconciliation raises IEMOCAP speaker attribution accuracy from 85.0% to 95.0%, and EMA updates improve accuracy on all three datasets.Table 6 reports speaker attribution accuracy after optimal permutation matching, with macro ARI ×100 in parentheses.Source: §D.1 · p. 21, Table 6 · p. 21

D.2 Human Audit of LLM-Judge Scores

  • Across 200 sampled responses, aggregate LLM and human means are 84.6 and 86.1, respectively, with absolute gaps within 1.8 points.Similar aggregate means do not establish item-level agreement between human and automatic scores.Source: §D.2 · p. 21, Table 7 · p. 21

E Limitations

VoxPolyBench uses generated dialogues and synthesized speech, with human review to control question and evidence quality. Extending question generation and validation to real long conversations with original audio remains future work.

E Limitations

  • VoxPolyBench uses generated dialogues and synthesized speech, with human review of event plans and QA pairs.The construction uses planned events and human validation to control question and evidence quality.Source: E Limitations · p. 21
  • Automatically generating challenging multi-hop questions from long conversations makes reliable links between dispersed evidence and unambiguous answers difficult to ensure.This motivates the benchmark’s planned events and human validation.Source: E Limitations · p. 21
  • Future work will explore question generation and validation over transcripts of real long conversations paired with original audio.The stated aim is to broaden coverage of complex reasoning and real-world spoken interactions.Source: E Limitations · p. 22
Loading 2609.32522v1…