PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Memory Majority: Latent-Source Reasoning for Multi-Agent Memory Arbitration

Chenchen Lin, Wenhao Yuan, Xuehe Wang, Edith Cheuk Han Ngai

arXiv:2608.19701v1cs.AI

TL;DR

Existing memory arbitration often treats retrieved entries as independent, despite query-specific correlations and gaps in retrieved evidence. CAMA models those dependencies and recovers additional evidence, and experiments report that it outperforms state-of-the-art baselines while suppressing correlation-induced false majorities.

  • Problem

    Existing methods commonly treat retrieved memories as independent evidence, while query-conditioned correlations and missing independent evidence remain insufficiently addressed.

    Source: Evidence Aggregation and Recovery · p. 2

  • Method

    CAMA models query-conditioned evidence dependencies with neural inference and provenance-based priors, arbitrates over latent evidence factors, and sequentially recovers missing evidence.

    Source: Introduction · pp. 1–2

  • Results

    Across multiple benchmarks, CAMA outperforms state-of-the-art baselines and suppresses false majorities induced by correlated memories.

    Source: Introduction · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    The paper concludes that effective memory reasoning requires understanding underlying dependencies, not merely accumulating more memories.

    Source: Conclusion · p. 8

    Candidate extraction is orthogonal to CAMA, which focuses on evidence modeling and memory arbitration.

    Source: Problem Formulation · p. 2

Abstract

from arXiv · show

Long-term multi-agent systems continuously accumulate the memories produced by different agents. Existing memory methods typically treat retrieved memories as independent evidence and combine them through voting or weighting. However, this independence assumption often fails in multi-agent settings: memories written by different agents may inherit the same upstream source or shared bias, causing correlated evidence to be repeatedly counted and creating a false majority. We term this failure mode \textit{Memory Correlation Bias}. To address the issue, we propose the \textbf{C}orrelation-\textbf{A}ware \textbf{M}emory \textbf{A}rbitration (CAMA) framework that jointly decouples retrieved memories and recovers missing independent evidence. We model the retrieved memories as query-conditioned evidence groups and combine neural dependency inference with provenance-based symbolic priors to estimate the effective number of independent evidence sources, thereby preventing correlated memories from forming a false majority. Since critical independent evidence may be absent from the initial retrieval set, \textsc{CAMA} further learns a sequential recovery policy that actively retrieves alternative evidence or traces upstream sources before making the final decision, aiming to recover sufficient independent evidence for reliable arbitration while minimizing retrieval cost. Experiments on multiple benchmarks demonstrate the superiority of our method over the state-of-the-art baseline methods, suppressing false majorities induced by correlated memories.

Introduction

CAMA addresses false majorities caused by correlated memories and missing independent evidence. It models query-conditioned evidence dependencies, arbitrates at the evidence-factor level, and actively recovers evidence when needed.

Introduction

  • Memory Correlation Bias arises when correlated memories inflate hypothesis support; writing a false majority back to shared memory can amplify errors in later decisions.Existing methods often assume retrieved memories are independent, while query-specific redundancy and missing independent evidence remain unresolved.Source: Introduction · p. 1, Evidence Aggregation and Recovery · p. 2
  • CAMA jointly decouples correlated memories, arbitrates using latent evidence factors, and recovers missing independent evidence while minimizing retrieval cost.Its approach combines neural dependency inference with provenance-based symbolic priors.Source: Introduction · pp. 1–2
  • Across multiple benchmarks, CAMA outperforms state-of-the-art baselines and suppresses false majorities induced by correlated memories.The conclusion also reports favorable efficiency.Source: Introduction · p. 2

Query-Conditioned Evidence Decoupling

Effective Independent Evidence Estimation

  • The effective number of independent sources is estimated from slot-level presence, so repeated memories of one factor do not increase evidence mass.A Hill diversity measure summarizes the relative presence of latent evidence slots.Source: Effective Independent Evidence… · p. 4

Factor-Level Conflict Arbitration

  • CAMA aggregates hypothesis support by evidence slot and uses source reliability attributes, including direct-observation status, agent reliability, and upstream source type.Normalized aggregation prevents memory multiplicity from biasing estimated reliability.Source: Factor-Level Conflict Arbitration · p. 4

Active Independent-Evidence Recovery

  • When evidence is insufficient or dependencies remain unresolved, recovery can expand retrieval with reformulated queries, trace memories to parent sources, or stop.If provenance is unavailable, the process uses Expand and Stop; stopping requires sufficient independent evidence and a concentrated arbitration posterior, or exhaustion of the recovery budget.Source: Active Independent-Evidence Recovery · pp. 4–5

Evidence-Guided Recovery Optimization

  • The recovery policy balances evidence acquisition against access costs and learns action value from long-term returns, with a terminal reward tied to arbitration correctness.Nonterminal recovery actions incur memory-access costs.Source: Evidence-Guided Recovery Optimization · p. 5

Experiments

Experiments evaluate CAMA on three long-term memory benchmarks and report stronger performance, correlation-bias mitigation, and a favorable accuracy–efficiency trade-off. Ablations and hyperparameter analyses examine the contributions and robustness of its components.

Experimental Setup

  • The evaluation uses MemoryAgentBench, LongMemEval, and LoCoMo with correlation-aware variants augmented by correlated and complementary evidence entries.Baselines include retrieval, memory-management, and multi-agent methods; experiments use two backbone LLMs with shared settings across methods.Source: Experimental Setup · pp. 5–6
  • Evaluation covers benchmark task performance and four correlation-bias metrics: CMR, RS, IEG, and ERR.The task metrics include Fact Consolidation scores on MemoryAgentBench and EM, F1, BERTScore, and judge-based scores on LongMemEval and LoCoMo.Source: Experimental Setup · p. 6

Experimental Results

  • CAMA consistently achieves superior performance across benchmarks and backbones in the overall comparison.Table 1 presents the comparison across three benchmark datasets.Source: Experimental Results · p. 6, Table 1 · p. 6
  • CAMA consistently improves correlation-aware metrics, mitigating memory correlation bias through evidence decoupling, provenance modeling, and evidence recovery.Table 2 evaluates correlation-bias mitigation under DeepSeek-V4-Flash.Source: Table 2 · p. 6, Experimental Results · p. 7
  • Removing any CAMA component degrades performance, with decoupling and provenance supporting dependency identification and recovery acquiring missing independent evidence.The ablation study covers MemoryAgentBench and LongMemEval under DeepSeek-V4-Flash.Source: Experimental Results · p. 7, Table 3 · p. 7
  • CAMA remains robust across hyperparameter settings; increasing K can add redundancy, while increasing B improves evidence recovery with diminishing returns.The analysis also describes J as balancing factor separation against fragmentation and moderate µ as balancing provenance guidance against assignment flexibility.Source: Experimental Results · p. 7, Figure 2 · p. 7
  • CAMA achieves a favorable accuracy–efficiency trade-off by activating recovery selectively when retrieved evidence is insufficient.∆Acc./kToken measures accuracy improvement over Vanilla RAG per thousand consumed tokens.Source: Experimental Results · p. 7, Table 4 · p. 7

Conclusion

The conclusion identifies false majorities from correlated memories as a challenge in long-term multi-agent systems. It presents CAMA’s latent-factor arbitration and selective evidence recovery, reporting stronger performance and favorable efficiency.

Conclusion

  • Correlated memories may create false majorities and persistent reasoning errors in long-term multi-agent systems.The conclusion frames this as Memory Correlation Bias.Source: Conclusion · pp. 7–8
  • CAMA evaluates evidence at the level of latent independent factors and selectively recovers missing evidence when current memories are insufficient.The framework combines neuro-symbolic evidence modeling, factor-level arbitration, and adaptive recovery.Source: Conclusion · p. 8
  • Experiments show that CAMA consistently outperforms existing approaches with favorable efficiency.The conclusion states that reliable memory reasoning requires understanding dependencies, not only accumulating more memories.Source: Conclusion · p. 8

Algorithm

CAMA iteratively updates hypotheses, evidence assignments, and arbitration using the current memory slice and provenance, then stops under budget, sufficiency, or policy conditions. Its evaluation metrics are defined over constructed cases containing correlated or independent evidence.

Inference Procedure and Complexity

  • At each recovery step, CAMA updates hypotheses, evidence assignments, and the arbitration posterior from the current memory slice and provenance structure.The final output includes the selected hypothesis and factor-level attribution.Source: Inference Procedure and Complexity · p. 10
  • Inference stops when the recovery budget is exhausted, evidence sufficiency is reached, or the learned policy selects Stop.After a nonterminal action, CAMA recomputes latent evidence factors and arbitration results from the updated memory slice and provenance.Source: Inference Procedure and Complexity · p. 10

Correlation-Aware Evaluation Metrics

  • The correlation-aware metrics are defined over evaluation cases whose retrieved memories may share an evidence source or come from distinct sources.The passage denotes the model predictions before and after CAMA as y and y*, respectively.Source: Correlation-Aware Evaluation Metrics · p. 10

Query-conditioned Evidence Decoupling

CAMA assigns retrieved memories softly to query-dependent latent evidence factors, using their assignments to represent dependencies and redundancy. Provenance information can guide those assignments as an auxiliary prior without enforcing hard constraints.

Query-conditioned Evidence Decoupling

  • CAMA infers a soft assignment matrix between retrieved memories and latent evidence factors, whose rows represent memory contributions and whose columns represent query-dependent factors.The factors support or contradict candidate hypotheses.Source: Query-conditioned Evidence Decoupling · p. 10
  • Memories assigned to the same factor count as redundant evidence, while memories assigned to different factors provide potentially independent support.Assignment overlap models dependency under the current query, and larger overlap indicates stronger evidential redundancy.Source: Query-conditioned Evidence Decoupling · p. 10
  • Provenance, when available, guides evidence assignment as an auxiliary structural prior without imposing hard dependency constraints.The resulting factor representation supports effective independent evidence estimation and arbitration.Source: Query-conditioned Evidence Decoupling · p. 10

Active Independent-Evidence Recovery

CAMA treats the initial retrieval as a partial view that may overrepresent correlated evidence or omit evidence needed for reliable arbitration. It iteratively refines that view before deciding.

Active Independent-Evidence Recovery

  • The initial memory slice may overrepresent existing evidence factors because of hidden correlations or omit critical independent factors needed for a decision.Either gap can leave the retrieved evidence insufficient for reliable arbitration.Source: Active Independent-Evidence Recovery · p. 10
  • After each recovery action, CAMA updates the memory slice and re-estimates evidence diversity, assignment confidence, and arbitration uncertainty.The updated structure feeds subsequent arbitration and recovery decisions.Source: Active Independent-Evidence Recovery · p. 11
  • CAMA chooses among Expand to find missing independent factors, Trace to investigate provenance dependencies, and Stop when independent support and arbitration reliability are sufficient.The policy selects an action based on whether the current state calls for more evidence, dependency investigation, or termination.Source: Active Independent-Evidence Recovery · p. 11

Detailed Dataset Descriptions

The evaluation spans three long-term memory benchmarks and adds correlation-aware variants that preserve task answers while varying evidence composition and provenance.

Benchmark coverage

  • The evaluation covers MemoryAgentBench, LongMemEval, and LoCoMo, spanning dynamic memory management, long-horizon retrieval, and historical-conversation reasoning.The benchmarks represent different aspects of memory-augmented LLM agents.Source: Detailed Dataset Descriptions · p. 11

MemoryAgentBench

  • MemoryAgentBench tests incremental multi-turn memory updates, retrieval, and management, including conflicting updates and selective forgetting.Its evolving interactions can make historical memories outdated, redundant, or inconsistent.Source: Detailed Dataset Descriptions · p. 11

LongMemEval

  • LongMemEval tests retrieval and reasoning over long, multi-session conversation histories where relevant information is distributed across interactions.Repeated interactions may produce entries describing similar user states or events.Source: Detailed Dataset Descriptions · p. 11

LoCoMo

  • LoCoMo tests reasoning over multi-session dialogues with evolving user states and contains records that may share an underlying event or user state.This makes it a testbed for correlation-aware arbitration in long-term conversational agents.Source: Detailed Dataset Descriptions · p. 11

Correlation-aware variants

  • Correlation-aware variants add memories derived from shared sources and distinct independent factors while keeping original queries and ground-truth answers unchanged.Correlated entries use controlled paraphrasing or summarization with recorded provenance; the construction changes evidence composition and multiplicity to test responses to correlation and source diversity.Source: Detailed Dataset Descriptions · pp. 11–12

Comparison Baselines

The baselines represent direct aggregation, memory organization and retrieval, and multi-agent reasoning. CAMA instead models evidence-factor correlations and actively recovers missing independent evidence.

Retrieval-based aggregation

  • Vanilla RAG directly supplies retrieved memories as generation context without explicitly modeling their dependencies or reliability.It serves as a baseline for assessing correlation-aware evidence arbitration.Source: Comparison Baselines · p. 12
  • Majority Voting selects the hypothesis supported by the most memory entries, implicitly treating each entry as an independent vote.The description notes that false majorities may arise when entries share an underlying source.Source: Comparison Baselines · p. 12

Memory management and retrieval

  • Mem0 organizes long-term memories through extraction, addition, updating, and consolidation.It is included to test whether memory management can mitigate correlation issues through consolidation.Source: Comparison Baselines · p. 12
  • HippoRAG organizes memories as interconnected structures and uses knowledge graphs to support long-range retrieval and multi-hop reasoning.It exploits structural relationships but does not explicitly arbitrate evidence-level correlations.Source: Comparison Baselines · p. 12

Multi-agent reasoning

  • MAD uses iterative agent debate, while MADAM-RAG coordinates agents for retrieval and aggregation under conflicting evidence.MADAM-RAG focuses more directly on retrieval-level conflict resolution than MAD.Source: Comparison Baselines · p. 12

Baseline overview

  • CAMA models memory correlations at the evidence-factor level and actively recovers missing independent evidence before arbitration.The comparison spans retrieval-based aggregation, long-term memory management, and multi-agent reasoning approaches.Source: Comparison Baselines · p. 12

Metric Descriptions

The evaluation pairs benchmark task performance with four metrics targeting arbitration when memories are correlated or independent evidence is missing.

Task-Level Performance

  • Task-level evaluation reports MemoryAgentBench Fact Consolidation scores and LongMemEval and LoCoMo answer correctness, semantic similarity, and response-quality metrics.MemoryAgentBench includes Single-Hop Fact Consolidation, Multi-Hop Fact Consolidation, and an overall score; the other benchmarks use EM, F1, BERTScore, and judge-based evaluation.Source: Task-Level Performance · p. 12

Memory Correlation Bias Evaluation

  • CMR measures how often the final prediction stays correct when correlated entries supporting an incorrect hypothesis outnumber independent memories supporting the correct answer.It evaluates cases with minority correct evidence.Source: Memory Correlation Bias Evaluation · p. 12
  • RS measures decision sensitivity to replicating correlated memories from the same source; lower RS indicates less sensitivity to redundant replication.Source: Memory Correlation Bias Evaluation · p. 13
  • IEG measures performance improvement after adding independent evidence, with larger gains indicating better use of complementary evidence from distinct sources.Source: Memory Correlation Bias Evaluation · p. 13
  • ERR measures the proportion of evidence-insufficient initial cases that a method correctly resolves with its final prediction.The evaluated subset is designated by benchmark metadata and lacks at least one query-relevant independent factor initially; CAMA predicts after recovery, while methods without recovery use the initial slice.Source: Memory Correlation Bias Evaluation · p. 13

Details of the Ablation Study

The ablation variants remove individual modules to test their roles in correlation-aware modeling, evidence recovery, and recovery control.

Details of the Ablation Study

  • Without evidence decoupling, arbitration treats each retrieved memory entry as an independent source instead of modeling latent evidence factors.The variant removes the memory-to-factor assignment and effective independent evidence estimation.Source: Details of the Ablation Study · p. 13
  • Without the provenance prior, latent evidence factors rely on query-conditioned memory representations alone, without provenance relations.The remaining evidence estimation, recovery, and arbitration components are unchanged.Source: Details of the Ablation Study · p. 13
  • Without Expand, recovery cannot acquire additional memories from alternative retrieval views to fill gaps in the initial set.The policy can still choose Trace or Stop.Source: Details of the Ablation Study · p. 13
  • Without Trace, recovery can expand retrieval but cannot follow provenance relations to investigate upstream dependencies.The policy can still choose Expand or Stop.Source: Details of the Ablation Study · p. 13
  • Without the learned policy, a fixed heuristic selects recovery actions instead of adapting them to balance evidence gains against recovery cost.Evidence decoupling and the recovery action space remain in place.Source: Details of the Ablation Study · p. 13

Additional Experimental Results

CAMA improves correlation-bias mitigation across benchmarks and metrics; ablations report complementary module contributions, while efficiency results describe its accuracy–cost trade-offs.

Memory Correlation Bias Mitigation

  • CAMA achieves the best performance across benchmarks and metrics, improving CMR while reducing RS; active recovery also improves IEG and ERR.Aggregation methods such as Vanilla RAG and Majority Voting show low CMR and high RS; Table 5 covers correlation-bias mitigation.Source: Memory Correlation Bias Mitigation · pp. 13–14, Table 5 · p. 14

Ablation Study

  • Removing any component degrades task and correlation-aware metrics, with evidence decoupling causing the largest drop, especially on CMR and RS.The provenance prior, Expand, Trace, and learned policy each address distinct dependency-modeling or recovery roles.Source: Table 6 · p. 14, Ablation Study · p. 14

Efficiency Analysis

  • CAMA uses fewer LLM calls and tokens than MAD and MADAM-RAG while achieving higher accuracy improvement per token.Table 7 evaluates computational efficiency on MemoryAgentBench.Source: Table 7 · p. 14, Efficiency Analysis · p. 14
  • CAMA adds moderate latency versus lightweight retrieval methods, while controlled context length and token use reflect selective rather than indiscriminate evidence expansion.The reported accuracy–efficiency trade-off includes costs for evidence decoupling and adaptive recovery.Source: Efficiency Analysis · pp. 14–15

Prompt Used

The prompts cover candidate-hypothesis generation, evidence evaluation, and retrieval-query expansion to seek complementary evidence.

Prompt Used

  • Candidate-hypothesis prompts generate distinct plausible conclusions from retrieved memories without ranking them or adding unsupported information.They allow multiple hypotheses when evidence is ambiguous and can draw on different subsets of memories.Source: Prompt Used · p. 15
  • Evidence-evaluation prompts assess a memory's relevance and whether it supports or contradicts a candidate hypothesis, including directness and evidential strength.Source reliability and evidence quantity are modeled separately.Source: Prompt Used · p. 15
  • Expansion prompts generate bounded alternative queries aimed at missing or underrepresented evidence while avoiding paraphrases of available memories.They target complementary independent evidence and alternative perspectives.Source: Prompt Used · p. 15
Loading 2608.19701v1…