PaperlayerView exact revision on arXiv

Source-linked AI summary

MMA: Multimodal Memory Agent

Yihao Lu, Wanru Cheng, Zeyu Zhang, Hao Tang

arXiv:2602.16493v1cs.CV

TL;DR

Long-horizon multimodal agents must reason over memories that differ in credibility, recency, and consistency, yet insufficient evidence can still yield confident errors. MMA scores memory reliability dynamically and introduces MMA-Bench to test belief dynamics under controlled multimodal conflict. Across FEVER, LoCoMo, and MMA-Bench, the framework improves stability, selective behavior, and diagnostic performance, while exposing a Visual Placebo Effect.

  • Problem

    Many memory systems treat retrieved items as equally reliable despite differences in source credibility, temporal recency, and consistency, creating a need for evidence-sensitive reasoning and abstention.

  • Method

    MMA assigns memory-level confidence using source credibility, temporal decay, and conflict-aware network consensus, then reweights evidence and supports selective abstention; MMA-Bench controls reliability priors and text–vision conflicts.

  • Results

    MMA improves reliability across FEVER, LoCoMo, and MMA-Bench, including 35.2% lower FEVER accuracy standard deviation and 41.18% Type-B accuracy in MMA-Bench Vision mode.

  • Takeaways & Limitations

    Explicit reliability modeling turns passive memory into epistemic filtering, improving stability and calibrated abstention while diagnosing and mitigating the Visual Placebo Effect.

  • Takeaways & Limitations

    As a post-retrieval module, MMA cannot recover relevant evidence that the underlying RAG system fails to retrieve, and strict consensus can be conservative in sparse information environments.

Abstract

from arXiv · show

Long-horizon multimodal agents depend on external memory; however, similarity-based retrieval often surfaces stale, low-credibility, or conflicting items, which can trigger overconfident errors. We propose Multimodal Memory Agent (MMA), which assigns each retrieved memory item a dynamic reliability score by combining source credibility, temporal decay, and conflict-aware network consensus, and uses this signal to reweight evidence and abstain when support is insufficient. We also introduce MMA-Bench, a programmatically generated benchmark for belief dynamics with controlled speaker reliability and structured text-vision contradictions. Using this framework, we uncover the "Visual Placebo Effect", revealing how RAG-based agents inherit latent visual biases from foundation models. On FEVER, MMA matches baseline accuracy while reducing variance by 35.2% and improving selective utility; on LoCoMo, a safety-oriented configuration improves actionable accuracy and reduces wrong answers; on MMA-Bench, MMA reaches 41.18% Type-B accuracy in Vision mode, while the baseline collapses to 0.0% under the same protocol. Code: https://github.com/AIGeeksGroup/MMA.

1 Introduction

Long-horizon memory agents face unreliable, stale, and conflicting evidence that can produce overconfident errors. MMA addresses this with dynamic reliability scoring, MMA-Bench, and risk-aware evaluation across established and diagnostic benchmarks.

  • Motivation: Retrieved memories vary in source credibility, recency, and consistency, but many systems implicitly treat them as equally reliable.This can propagate low-quality information through multi-step inference and obscure evidential uncertainty.
  • MMA: MMA reweights retrieved memories using source credibility, temporal decay, and conflict-aware network consensus, with selective prediction for insufficient support.Its confidence-aware memory layer prioritizes credible evidence and can flag low-confidence items for abstention.
  • MMA-Bench: MMA-Bench evaluates belief dynamics under controlled reliability priors and structured multimodal conflicts, including calibrated abstention.The benchmark is programmatically generated to stress long-horizon belief revision and epistemic behavior.
  • Findings: The study diagnoses a Visual Placebo Effect in which ambiguous visual inputs can induce unwarranted certainty in retrieval-augmented agents.This finding connects multimodal conflict evaluation with the need for epistemic awareness and justified abstention.
  • Evaluation: 35.2% lower accuracy standard deviation on FEVER accompanies matched baseline raw accuracy, while evaluation also reports fewer LoCoMo wrong answers and 41.18% Type-B Vision accuracy on MMA-Bench.The FEVER comparison is 59.93% vs. 59.87% raw accuracy, with standard deviations of ±1.62% vs. ±2.50%.

2 Related Work

Related work develops memory retrieval, management, uncertainty estimation, and selective prediction for long-context agents. Existing benchmarks, however, generally do not jointly control evolving source reliability and structured cross-modal conflict.

  • Memory-Augmented LLM Agents: Memory-augmented agents extend long-horizon interaction through external-memory writing and retrieval, with structured memory, paging, hierarchies, and lifecycle operations.These systems improve the retrieval-and-inject pipeline through increasingly structured memory management.
  • MMA-Bench: MMA-Bench complements prior suites by controlling source reliability priors and pairing multimodal evidence for diagnosis under conflict.Its design targets belief dynamics and epistemic behavior rather than only static long-context performance.
  • Uncertainty and Selective Prediction: Uncertainty and self-consistency methods motivate selective prediction approaches such as conformal language modeling and conformal abstention.Related analyses also examine how standard training and evaluation can incentivize overconfidence.
  • Benchmarks: Long-context and memory-centric benchmarks primarily assess retrieval, correctness, or static consistency rather than belief revision under drifting evidence and modality disagreement.The cited discussion identifies a gap in evaluating whether agents commit, hedge, or defer when evidence quality changes.

3 The Proposed Method And Benchmark

MMA extends a memory-augmented agent with a confidence module that reweights retrieved memories using source reliability, temporal decay, and network consensus. MMA-Bench evaluates multimodal belief dynamics through controlled conflicts, layered probes, and cognitive-dynamics metrics.

  • 3.2 Multimodal Memory Agent (MMA): MMA extends MIRIX with a confidence module that scores retrieved memories and supports reweighting, reasoning modulation, and abstention.The score combines source credibility, temporal decay, and conflict-aware network consensus.
  • 3.2 Multimodal Memory Agent (MMA): Source reliability assigns trustworthiness priors, while temporal decay models information aging and network consensus filters semantic support and contradictions.Positive support factors reinforce confidence; negative factors penalize contradictions.
  • 3.3 MMA-Bench: MMA-Bench compares Text Mode with oracle captions against Vision Mode with raw images and uses risk-aware betting to reward justified abstention and penalize overconfidence.The benchmark targets multimodal conflict and epistemic calibration rather than accuracy alone.
  • 3.3 MMA-Bench: Each benchmark case spans 10 temporal sessions, establishing speaker reliability, injecting adversarial noise, creating a multimodal conflict, and resolving or withholding the ground truth.The logic matrix categorizes conflicts by source reliability and visual evidence.
  • 3.3 MMA-Bench: The benchmark evaluates foundational QA, a 3-step belief probe with CoRe scoring, and cognitive dynamics including modality alignment, relative reasoning uncertainty, and belief revision stability.Type B treats visual evidence as implying TRUE, whereas Type C/D treat it as implying UNKNOWN.
  • 3.3 MMA-Bench: Self-Correction Rate measures correction after an initial error, while False Confession Rate diagnoses abandoning correct beliefs under reflection pressure.A high FCR relative to SCR indicates prompt-induced skepticism rather than genuine epistemic calibration.

4 Experiments

Experiments evaluate MMA on text-centric and adversarial multimodal benchmarks, showing improved stability, selective behavior, and conflict handling. Ablations indicate that source reliability, consensus, and temporal decay each address distinct failure modes.

  • FEVER and benchmark results: 35.2% lower accuracy standard deviation on FEVER accompanies matched baseline accuracy, indicating improved stability across seeds.MMA records ±1.62% variance versus ±2.50% for the baseline.
  • MMA-Bench results: 41.2% Vision Dominant accuracy in MMA-Bench Type B contrasts with the baseline’s 100% Confusion rate, indicating active conflict resolution.The baseline defaults to “Unknown” in high-noise reliability-inversion scenarios.
  • MMA-Bench results: MMA identifies information gaps in Text Mode, whereas the baseline’s high raw accuracy in indeterminate scenarios can result from defaulting to “Unknown”.The analysis characterizes MMA’s behavior as intentional prudence and the baseline’s as retrieval limitation.
  • Cognitive analysis: 100% of 62 analyzed erroneous instances fall into “Logic Collapse”, where agents admit error during reflection but fail to update the initial verdict.Both MIRIX and MMA record a numeric FCR of 0%, despite different behavior from the foundation model.
  • Ablation study: Removing source reliability yields 0.0% accuracy in deterministic MMA-Bench Type A/B scenarios, reflecting mechanical defaulting to “Unknown”.The ablation links this failure to the absence of a prior trust distribution.
  • Ablation study: Removing consensus produces a −0.69 Type D score, while restoring consensus buffers the drop to −0.38 by filtering unsupported hallucinations.The comparison highlights consensus as a safety buffer against multimodal noise.
  • Ablation study: Removing temporal decay causes 0.0% Type A accuracy in MMA-Bench Vision Mode, showing that historical noise overwhelms the signal without temporal awareness.The failure is reported specifically when visual features compound cross-modal instability.

5 Conclusion

MMA reframes passive memory storage as active epistemic filtering through dynamic reliability scoring and calibrated abstention. Across FEVER, LoCoMo, and MMA-Bench, the framework improves stability and supports more prudent multimodal decision-making.

  • Conclusion: MMA-Bench identifies the Visual Placebo Effect, in which multimodal agents inherit latent visual bias from foundation models.MMA is reported to mitigate this bias and restore decision-making agency in deterministic scenarios.
  • Conclusion: MMA achieves a superior risk-coverage trade-off, delivering high utility in safety-critical environments.The conclusion frames reliability modeling as a cognitive guardrail for high-stakes applications.

Limitations

The paper identifies two boundaries for MMA: it depends on upstream retrieval recall and may become conservative when consensus is enforced in sparse information environments.

  • Limitations: MMA cannot recover relevant evidence that the underlying retrieval system fails to retrieve, although it can filter retrieved hallucinations.This limitation follows from MMA’s role as a post-retrieval module.
  • Limitations: Strict consensus enforcement can be conservative in low-density information environments, motivating adaptive gating based on context entropy.The limitation is supported by the paper’s LoCoMo analysis.

B Results on FEVER Benchmark

On FEVER, MMA preserves baseline raw accuracy while improving robustness and abstention calibration. Its selective advantage becomes clearer as abstention receives greater reward, with lower-risk answers at reduced coverage.

  • Overall Performance and Stability: ≈59.9% raw accuracy is comparable for the baseline and MMA, while MMA reduces variance from ±2.50% to ±1.62%.The evaluation uses the first 500 samples across three random seeds.
  • Prudence and Calibration: MMA correctly abstains on 103.7 NEI cases on average, compared with 100.7 for the baseline.MMA abstains more often overall: 226.3 versus 221.0 times per 500 samples.
  • Prudence and Calibration: As abstention reward α increases, MMA’s Selective Score consistently rises above the baseline while both models begin near ≈59.9% accuracy at α = 0.The comparison evaluates risk-sensitive utility under varying abstention incentives.
  • Risk-Coverage Trade-off: MMA trades lower coverage for lower risk, with its points clustering toward the bottom-left relative to the baseline in the risk-coverage plot.The reported mechanism filters low-confidence retrieval results through consensus.
  • LoCoMo: 79.64% Actionable Accuracy and 883.6 Utility are achieved by the LoCoMo ‘st’ variant, slightly surpassing the baseline in sparse contexts.The result demonstrates configuration-dependent adaptation to information density.

Appendix C.

The FEVER ablation evaluates MMA across three random seeds and shows that temporal decay and consensus improve prudence and stability, while source reliability matters most in adversarial settings.

  • Experimental setup: FEVER ablations evaluated variants on 500 examples across three random seeds.The variants are reported in Table 6 and Figure 7.
  • Temporal decay: 68.96% Actionable Accuracy is achieved by the no-time variant ‘cs’, alongside 95.0 Correct Abstentions.Removing temporal decay produces the lowest values for both measures.
  • Temporal decay: Without temporal awareness, ‘cs’ underperforms as the safe-abstention reward α increases.The authors associate this with failure to identify outdated information and overconfident hallucinations.
  • Network consensus: 71.61% Actionable Accuracy with ±0.43% standard deviation is achieved by full MMA, compared with 72.05% and approximately 2.34% for ‘st’.The no-consensus variant also has the highest Wrong Abstains at 131.0, indicating greater conservatism.
  • Source reliability: The no-source variant ‘tc’ achieves the highest raw FEVER accuracy, while source reliability becomes indispensable in adversarial mixed-reliability scenarios.The authors attribute the raw-accuracy result to FEVER’s homogeneous Wikipedia-based sources.
  • Overall result: Full MMA provides the reported trade-off between avoiding ‘cs’ blind guessing and ‘st’ erratic conservatism.The paper characterizes the resulting solution as stable, prudent, and trustworthy for fact verification.

C Results on LoCoMo Benchmark

On LoCoMo, the baseline leads in raw accuracy, but MMA’s safety-oriented configuration improves actionable utility and reduces wrong answers; component effectiveness depends on domain sparsity.

  • Evaluation: The LoCoMo evaluation compares MMA with the MIRIX baseline across reasoning dimensions.LoCoMo represents long-term conversational history with sparse information density and low adversarial conflict.
  • Main results: 77.37% Overall Accuracy and 573.5 Utility are achieved by the baseline, versus 72.31% and 488.0 for MMA.The baseline’s advantage is associated with aggressive retrieval and 97.73% coverage.
  • Category-wise analysis: 62.31% Multi-Hop accuracy for MMA trails the baseline’s 76.01%, while temporal performance is 77.05% versus 78.00%.The consensus module may overly penalize weak but valid multi-hop links in sparse narratives.
  • Safety and robustness: 609.0 Utility is achieved by the ‘st’ configuration, which also has the lowest wrong-answer count and surpasses the baseline in utility.The configuration removes consensus while retaining Source and Time components.
  • Ablation: 79.64% Actionable Accuracy is achieved by ‘st’, which the LoCoMo results identify as the most reliable safety-oriented configuration.Table 8 likewise identifies ‘st’ as having the best safety profile.
  • Consensus ablation: 298 Wrong Answers and 609.0 Utility are achieved by ‘st’ after removing consensus.The paper attributes this to sparse, thematically related but factually irrelevant retrieved neighbors introducing consensus noise.
  • Source ablation: 344 Wrong Answers and 471.5 Utility result from removing Source in ‘tc’.The finding underscores source credibility’s role in filtering misleading persona-driven dialogue context.

D.1 Analysis of Foundation Models

Foundation models show strong retrieval and comprehension but weak epistemic arbitration when text and vision conflict. MMA-Bench exposes visual bias, overconfidence, and unstable self-correction.

  • Evaluation setup: GPT-4.1-mini and Qwen3-VL-Plus receive full dialog-history access, isolating belief reasoning from retrieval limitations.Despite this advantage, the probes reveal significant deficits in belief dynamics.
  • Perception and arbitration: Strong fact retrieval and distraction filtering coexist with a significant drop in three-step verdict accuracy.The results distinguish reading comprehension from epistemic arbitration.
  • Modality preference: 82.4% visual-signal alignment is observed for Qwen3-VL-Plus, compared with 64.7% for GPT-4.1-mini in Type B inversion scenarios.Both models struggle to prioritize objective visual evidence over a historically reliable textual source.
  • Visual placebo effect: GPT-4.1-mini’s Type D CoRe score drops from 0.85 in text mode to 0.23 in vision mode.The paper terms this visual placebo effect: ambiguous or irrelevant images can induce definitive answers instead of prudence.
  • Visual placebo effect: Figure 10 compares the models across Text (Oracle) and Vision (Raw) modes, separating Core Acc from CoRe Score.The figure frames visual inputs as potential distractors and contrasts comprehension with epistemic prudence.
  • Self-correction: More than 50 cases involve models flipping from a correct verdict to an incorrect one during reflection.This qualitative instability accompanies high numerical self-correction rates.
  • Self-correction: The analysis attributes some self-correction to instructional sycophancy rather than authentic introspection.The models conform to skepticism encoded in the reflection prompt.
  • Self-correction: The acting and thinking systems can diverge, with models wagering on one verdict and immediately confessing another.The paper links this disconnect to immature coherent belief-state maintenance.

D.2 Ablation on MMA-Bench

MMA-Bench ablations show that Source and Time support decisive, stable behavior, while Consensus buffers visual placebo effects in indeterminate scenarios.

  • Setup: The ablation compares three variants with the full S + T + Ccon model on MMA-Bench.Table 5 and Figure 11 isolate the contributions of each component.
  • Source reliability: 0.0% deterministic accuracy occurs for ‘tc’ without Source, despite 1.0 Type D and 96.7% Type C scores.The pattern indicates defaulting to Unknown rather than exercising prudence; full MMA reaches 50.0% in Vision Type A.
  • Network consensus: 52.9% Type B accuracy is achieved by ‘st’ without Consensus, versus 41.2% for MMA.Its greater aggression toward visual evidence is paired with a Vision Type D score of −0.69, compared with −0.38 for MMA.
  • Network consensus: Consensus buffers the Type D drop by validating visual inputs against the semantic neighborhood.The full model is more conservative in Type B but safer in indeterminate contexts.
  • Temporal decay: 0.0% Vision Type A accuracy is achieved by ‘cs’ without Time, despite 40.0% Type A accuracy in Text Mode.Full MMA maintains approximately 50% Vision Type A accuracy, supporting temporal awareness for multimodal robustness.
Loading 2602.16493v1…