PaperlayerView exact revision on arXiv

Source-linked AI summary

MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

Xiyu Ren, Zhaowei Wang, Yiming Du, Zhongwei Xie, Chi Liu, Xinlin Yang, Haoyue Feng, Wenjun Pan, Tianshi Zheng, Baixuan Xu, Zhengnan Li, Yangqiu Song, Ginny Wong, Simon See

arXiv:2605.14906v1cs.CV

TL;DR

Existing benchmarks do not fairly compare long-context LVLMs with memory-augmented agents on multimodal questions requiring visual evidence. MEMLENS provides a length-controlled benchmark and finds complementary weaknesses: long-context LVLMs degrade with longer conversations, while memory agents lose visual fidelity during compression.

  • Problem

    Existing benchmarks lack a length-controlled comparison of long-context LVLMs and memory-augmented agents on questions requiring genuine visual evidence.

  • Method

    MEMLENS evaluates 789 cross-modal questions spanning five memory abilities across multimodal multi-session conversations and context lengths from 32K to 256K tokens.

  • Results

    Multi-session reasoning caps most evaluated systems below 30%, while long-context LVLMs degrade with growing conversations and memory agents remain length-stable but lose visual fidelity under storage-time compression.

  • Takeaways & Limitations

    The results support combining long-context attention with structured multimodal retrieval rather than scaling either component in isolation.

  • Takeaways & Limitations

    MEMLENS is a controlled diagnostic benchmark, not a training dataset, and its evaluation scope is multimodal long-term conversational memory.

Abstract

from arXiv · show

Memory is essential for large vision-language models (LVLMs) to handle long, multimodal interactions, with two method directions providing this capability: long-context LVLMs and memory-augmented agents. However, no existing benchmark conducts a systematic comparison of the two on questions that genuinely require multimodal evidence. To close this gap, we introduce MEMLENS, a comprehensive benchmark for memory in multimodal multi-session conversations, comprising 789 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal) at four standard context lengths (32K-256K tokens) under a cross-modal token-counting scheme. An image-ablation study confirms that solving MEMLENS requires visual evidence: removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images. Evaluating 27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs achieve high short-context accuracy through direct visual grounding but degrade as conversations grow, whereas memory agents are length-stable but lose visual fidelity under storage-time compression. Multi-session reasoning caps most systems below 30%, and neither approach alone solves the task. These results motivate hybrid architectures that combine long-context attention with structured multimodal retrieval. Our code is available at https://github.com/xrenaf/MEMLENS.

1 Introduction

MEMLENS addresses the lack of a length-controlled comparison between long-context LVLMs and memory-augmented agents on multimodal questions requiring visual evidence. Across 789 questions and four context lengths, its evaluation reveals distinct memory-ability performance and complementary limitations, motivating hybrid architectures.

  • Benchmark gap: No current benchmark length-controls comparisons between long-context LVLMs and memory-augmented agents on questions requiring visual evidence.Existing long-context multimodal benchmarks mainly evaluate LVLMs, while conversational memory benchmarks are text-only.
  • Benchmark design: MEMLENS contains 789 questions spanning information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal.The benchmark targets multimodal multi-session conversations and requires cross-modal reasoning over visual and textual evidence.
  • Evaluation: 27 LVLMs and 7 memory-augmented agents are evaluated at 32K, 64K, 128K, and 256K context lengths.The comparison covers both approaches across all four standardized context lengths.
  • Key findings: Multi-session reasoning caps most evaluated systems below 30%, and strong information extraction does not predict multi-session reasoning.The five memory abilities are described as largely independent.
  • Implications: Long-context LVLMs and memory-augmented agents exhibit complementary failure modes, motivating architectures that combine long-context attention with structured multimodal retrieval.The proposed direction avoids scaling either component in isolation.

2 Related Work

Prior work develops diverse memory representations and operations for text-only and multimodal LLM agents. Existing long-context benchmarks and multimodal extensions address retrieval, scaling, documents, videos, or multiimage inputs, but do not evaluate memory in multi-session conversations with visual evidence.

  • Memory-Augmented LLM Agents: Memory-augmented LLM agents use diverse representations and operations, including structured stores, paging, summarized retrieval, graphs, embeddings, notes, layered tiers, and sliding-window mechanisms.The passage describes these approaches across text-only memory agents and notes multimodal extensions, although the supplied text truncates before detailing them.
  • Long-Context and Conversational Memory Benchmarks: Most long-context benchmarks study retrieval and length scaling, while multimodal extensions cover documents, needle-style retrieval, long-form video, super-long documents, and multiimage inputs rather than multi-session conversations.The passage explicitly identifies neglect of the visual modality in core long-context benchmarks and a mismatch between existing multimodal inputs and conversational memory.

3 The MEMLENS Benchmark

MEMLENS is a 789-question benchmark for multimodal long-term conversational memory across four context lengths, five memory abilities, and interleaved image-text sessions. Its design makes visual evidence necessary for most questions through cross-modal reasoning over images and surrounding text.

  • Benchmark scope: 789 questions span standardized 32K/64K/128K/256K-token contexts and interleaved multimodal conversation sessions.The benchmark targets long-term conversational memory with questions requiring cross-modal reasoning over evidence images and surrounding textual context.
  • Memory abilities: Five abilities cover information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal.The abilities target retrieval, aggregation, temporal interpretation, evolving attributes, and calibrated refusal when supporting evidence is absent.
  • Cross-modal dependency: Correct answers for the four answerable types require joint visual-textual reasoning because evidence images contain information deliberately withheld from text.This cross-modal principle is empirically validated in the benchmark’s image-ablation analysis.
  • Dataset construction: The construction pipeline generates topic-grounded multimodal sessions, creates image-dependent questions, wraps evidence facts into complete sessions, and assembles timestamp-ordered histories.Haystack sessions are contextually related but uninformative, and four standardized context lengths are produced by varying their number.
  • Cross-modal dependency: 65.7% of questions are image-essential, 14.7% image-supportive, and 19.6% text-sufficient.Image-essential answers are unrecoverable without evidence images, while image-supportive images confirm or disambiguate textual facts; text-sufficient items include all answer-refusal questions and some textual MSR items.

4 Evaluation and Analysis

Evaluation across 27 LVLMs and seven memory-augmented agents shows complementary weaknesses: long-context LVLMs lose accuracy as visual evidence becomes harder to retrieve, while memory agents remain more length-stable but compress away visual fidelity. Multi-session reasoning remains the shared bottleneck, and current memory-agent training also substantially weakens answer refusal.

  • Evaluation setup: 27 LVLMs and seven memory-augmented agents are evaluated on MEMLENS, with Figure 2’s representative comparison covering 13 LVLMs and six agents.LVLMs use all 789 questions, while agents use a 195-question canonical subset.
  • Type-specific difficulty: 44.06% on MSR for Kimi-K2.5 and 32.17% for Gemini-3.1-Pro are the only results clearing 30%, making multi-session reasoning the shared capability ceiling.MSR requires cross-session aggregation over three to eight sessions; AR is easiest, with a 97.78% ceiling at 32K.
  • Answer refusal: 77.27% and 68.18% AR accuracy for frozen-backbone Mem0 and MemOS contrasts with 9–22% for agents that finetune their backbones, while M2A reaches 22.73% versus 81.82% under direct inference.The results suggest current RL/SFT reward design for memory management weakens abstention behavior.
  • Scaling trade-offs: ∼20% and ∼12% losses on IE and KU respectively show that LVLMs degrade on retrieval-heavy types as context grows, whereas memory agents exhibit a contrasting scaling profile.The MSR curve’s apparent flatness reflects a floor effect near 30%, not genuine robustness.
  • Error analysis: Nearly 90% of IE and KU errors are Visual, indicating failures to locate or read evidence images rather than failures to extract answers once images are reached.TR errors divide between Mixed and Reasoning, reflecting image-supportive grounding and small closed-set selection.

5 Conclusion

MEMLENS introduces a unified benchmark for multimodal conversational memory across LVLMs and memory-augmented agents. Results show persistent difficulty: LVLMs degrade with longer contexts, while memory agents remain length-stable but lose fine-grained visual evidence during storage.

  • Benchmark: MEMLENS is the first benchmark evaluating multimodal conversational memory for LVLMs and memory-augmented agents under a unified, length-controlled protocol.The protocol spans 32K–256K tokens and five memory abilities.
  • Results: 58.68% is the strongest LVLM’s accuracy at 32K, demonstrating that the task remains challenging even at the shortest evaluated context.The reported context-length range is 32K–256K tokens.
  • Results: LVLMs degrade sharply as contexts grow, whereas memory agents remain length-stable but discard fine-grained visual evidence during storage.The comparison identifies complementary weaknesses between direct long-context processing and memory-based storage.

Ethics Statement

MEMLENS is positioned as a controlled diagnostic benchmark rather than training data, with traceable releases and safeguards around image sourcing, annotation, and human review. Its evaluation items are protected from supervised-data exposure to preserve diagnostic value.

  • Benchmark purpose: MEMLENS is designed exclusively for evaluating multimodal long-term conversational memory under a unified, length-controlled protocol, not for training memory-equipped systems.Artifacts are released under frozen version tags so reported leaderboard cells remain traceable to exact items.
  • Data sourcing: 4,695 source images come from public web search using a non-person-centric ontology, excluding watermarks, stock-photo logos, and copyright overlays.Construction prompts do not ask models to identify, name, or describe depicted people.
  • Human review: Human review uses project members rather than crowd-workers and applies three sequential audits before items enter the released set.The audits assess cross-modal necessity, session naturalness and recoverability, and a stratified sample of haystack sessions.
  • Data-use restriction: 789 evaluation items should not be exposed as supervised data because doing so would compromise MEMLENS’s diagnostic value.The benchmark’s five abilities describe general functional requirements of long-horizon conversation.

Reproducibility Statement · B Evaluation Setup · B.1 Models

The evaluation combines API-based closed-source testing with locally served open-weight models and memory agents, using documented protocols, matched backbones, and adapters that preserve each system’s native input format. LVLMs and agents are compared across context lengths, with agents evaluated on a stratified subset because their pipelines are substantially slower.

  • Reproducibility Statement: Approximately $4,500 covered closed-source API evaluations and synthetic-data construction, including pipeline construction and benchmarking.The APIs used were provided by OpenAI, Anthropic, and Google.
  • Reproducibility Statement: Open-weight LVLMs were served locally with vLLM v0.17–0.18 on 8×A100-80GB nodes, using tensor parallelism for context windows at and above 128K.Appendix B documents decoding budgets, judge protocol, retrieval depth, and adapter details for four text-only pipelines.
  • B Evaluation Setup: 27 LVLMs and seven memory-augmented agent systems were evaluated, with LVLMs tested at 32K, 64K, and 128K and agents at all four context lengths.The LVLM context restriction reflects limited compute and incomplete 256K support.
  • B.1 Models: The seven memory agents comprised three multimodal systems—M3-Agent, M2A, and M3C—and four text-only systems, including Mem0, MemOS, MemAgent-7B, and Memory-T1.The multimodal systems use distinct retrieval or adaptation mechanisms, while the text-only systems use framework or recurrent-memory architectures as specified.
  • B.1 Models: Released checkpoints were used for RL-finetuned or LoRA-adapted agents, while Mem0 and MemOS both used Qwen3-8B for a matched-scale comparison.The matched backbone helps disentangle backbone quality from memory architecture.
  • B Evaluation Setup: All agents were evaluated on a stratified 195-question subset because agent pipelines are slower than direct LVLM inference, with M2A taking roughly 60× longer per question.The subset represents approximately one-quarter of the benchmark; agents included 256K evaluation, while LVLMs used 32K–128K.
  • B.1 Models: Text-only agents replaced every evidence image with BLIP-2 captions, M3-Agent used one composite image per session, and only M2A and M3C processed original interleaved messages directly.These adapters reflect each architecture’s supported input format.
  • B.1 Models: Direct LVLMs attended to original conversation pixels, whereas memory agents accessed transformed representations such as captions, composite images, or embedding-based entries without raw pixels at query time.The evaluation retained this asymmetry because each adapter matches the released checkpoint’s assumed input format.

B.2 Metrics · B.3 Infrastructure

MEMLENS uses LLM-as-Judge accuracy as its primary metric because heterogeneous answer formats undermine string matching, while Coverage and Per-Answer Accuracy expose answering trade-offs. Infrastructure combines vLLM with FlashAttention-2 on multi-GPU nodes, provider APIs, concurrent requests, and length-specific generation limits.

  • B.2 Metrics: LLM-as-Judge accuracy is the primary metric for all reported results, scored by canonical Qwen3-VL-235B-A22B-Instruct with thinking disabled.String-match metrics fail across heterogeneous formats and verbose rationales.
  • B.2 Metrics: Substring Exact Match checks whether a normalized reference answer appears in model output, but can inflate scores when reasoning traces mention the reference.The rule is generous to verbose models.
  • B.2 Metrics: Coverage measures the fraction of 699 answerable questions attempted, while Per-Answer Accuracy measures accuracy on attempted answers.These metrics complement overall accuracy by separating willingness to answer from correctness.
  • B.2 Metrics: Overall accuracy is recovered approximately as J ≈ (Cov × PA × 699 + ARcorrect) / 789, exposing the coverage–accuracy trade-off.The decomposition uses 699 answerable questions and 789 total benchmark questions.
  • B.3 Infrastructure: Local models run through vLLM v0.17–0.18 with FlashAttention-2 on 8×A100-80GB nodes, using tensor parallelism for 128K inputs.This describes the local-model serving configuration.
  • B.3 Infrastructure: API models use provider endpoints with 4–8 concurrent threads, while generation limits are 2,048 tokens for direct models and 16,384 for thinking models.The longer thinking-model limit accommodates reasoning traces.

C Dataset Construction Details … C.8 Conversation History Assembly

MEMLENS constructs cross-modal, multi-session evaluation instances by combining ontology-driven question generation, visually essential evidence, filtered web images, and long conversation histories. Its dataset design spans diverse visual formats and memory operations while preserving temporal structure for knowledge updates.

  • C.1 Problem Formulation: Each instance is a 4-tuple of timestamped multimodal sessions, a query, evidence-critical images, and a gold answer or NOT_MENTIONED.Systems must localize relevant sessions and ground answers in cross-modal reasoning over the critical image subset.
  • C.2 Topic Ontology: The ontology organizes four answerable abilities across identification, experience, and document tracks, expanding roughly 100 topics per ability into more than 12,000 subtopics.The tracks cover entities, everyday activities, and text-rich artifacts.
  • C.3 Subtype Detail: 65.7% of questions are image-essential, 14.7% image-supportive, and 19.6% image-nonessential.The taxonomy contains eight answerable subtypes plus Answer Refusal and isolates distinct visual or reasoning operations.
  • C.4 Question Construction Pipeline: Question construction proceeds through background generation, entity selection and image retrieval, entity abstraction, and question generation with quality verification.Abstraction removes entity names and uses under-specified anaphors so the image supplies entity identity while text supplies factual context.
  • C.4 Question Construction Pipeline: IE questions use image-based entity resolution followed by textual property retrieval, while MSR and TR include cross-session matching, counting, and timestamp- or image-based duration reasoning.A three-layer text-hackability defense is used for relevant cross-modal question routes.
  • C.5 Image Filtering: Every evidence and filler image passes relevance scoring and negative-content filtering, while global URL uniqueness and pHash deduplication prevent reused or near-duplicate images.Relevance requires CLIP above 0.30 and at least one secondary channel; pHash rejects candidates within Hamming distance 6.
  • C.6 Image Diversity: MEMLENS partitions images by dominant visual format into complementary perceptual regimes, with natural photographs supplying most IE, MSR, and KU evidence and documents testing dense visual text.This retrieval-driven organization avoids a narrow object-centric or OCR-only benchmark focus.
  • C.7 Image Sourcing, Licensing, and Release: All images come from public web searches, undergo artifact filtering, and receive provenance metadata; 4,695 unique images are distributed with the four context-length datasets.The release distinguishes author-produced artifacts from third-party image licenses and provides takedown and re-fetch support.

D Data Examples … E.1 Annotation Guidelines and Quality Assurance

MEMLENS illustrates five multimodal memory abilities through examples requiring visual grounding, cross-session retrieval, temporal inference, updates, or refusal. Its three-round quality-control process filters 20k candidates to 789 questions using evidence, ambiguity, difficulty, naturalness, and haystack checks.

  • D.1 Information Extraction: Information Extraction questions require identifying a visually grounded entity before retrieving its relevant fact or recalling an earlier image’s visual detail.Examples include entity abstraction, colors, counts, layouts, and on-screen text.
  • D.2 Multi-Session Reasoning: Multi-Session Reasoning questions aggregate prices, durations, or quantities, count matching sessions or items, and resolve whether cross-session references identify the same entity.At least one arithmetic operand may appear only in an image.
  • D.3 Temporal Reasoning: Temporal Reasoning questions compare time spans and establish chronology or absolute dates from textual dates, session timestamps, clock faces, or calendar images.Visual temporal cues can be the only available source for a date or time.
  • D.4 Knowledge Update: Knowledge Update questions follow a four-step preference chain anchored by different images, with the gold answer determined by the most recent state.Each step is grounded in a separate image.
  • D.5 Answer Refusal: Answer Refusal questions remove supporting evidence from the conversation history, making a refusal phrase rather than a content answer correct.The missing evidence is deliberate rather than accidental.
  • E.1 Annotation Guidelines and Quality Assurance: Round 1 independently reviews each of n = 20k generated candidates with two of four annotators for visual necessity, leakage, answer unambiguity, and calibrated difficulty.Items are accepted, revised and re-reviewed, or rejected; Cohen’s κ = 0.78 on n = 200 sampled items indicates substantial agreement.
  • E.1 Annotation Guidelines and Quality Assurance: Round 2 checks n = 2,145 evidence sessions for recoverable and distributed needle facts, natural dialogue, and adjacent image–text co-reference, regenerating failures.The sessions average 2.7 evidence sessions per question.
  • E.1 Annotation Guidelines and Quality Assurance: Round 3 audits 500/689 multimodal haystack sessions for topical relevance, absence of spurious answers, image quality, and natural conversational progression.Sessions failing checks are manually rewritten and refined.

E.2 Judge Validation Details

Judge validation finds strong agreement across independent judges and human consensus, with small, characterized leniency biases rather than systematic favoritism toward Qwen outputs. The evaluation corrects short-answer format bias and supplements human audits with deterministic rescoring of closed-form answers.

  • Cross-family validation: 96.40% item-level agreement and Spearman ρ = 0.97 show strong cross-family consistency between Qwen3-VL-235B and GPT-5.4-mini judges.The validation re-evaluated 800 items sampled from 73,784 judge calls; Cohen’s κ = 0.93 and p < 10−6 for the ranking correlation.
  • Human validation: 93.60% raw agreement with human consensus indicates high judge reliability, with errors strongly biased toward leniency: 29 false positives versus 2 false negatives.Three annotators independently labeled 484 items, resolving disagreements to consensus; Cohen’s κ = 0.86.
  • Family-bias check: +0.33% is the difference between Qwen-vs-GPT leniency gaps on Qwen-family versus non-Qwen outputs, indicating negligible practical self-favoritism.The gaps are +3.00% for Qwen-family outputs and +2.70% for non-Qwen outputs, an order of magnitude below the 3% practical-significance threshold.
  • Bias correction: 6 affected models receive corrected scores because the judge evaluates very short answers more leniently, creating format-dependent false positives.All scores reported in the paper use the corrected values.
  • Question-type validation: −6.50% is the largest cross-judge gap by question type, occurring on Information Extraction, while Multi-Session Reasoning shows zero gap and no type reverses the ranking.The Qwen3-VL-235B judge is equally or slightly more lenient than GPT-5.4-mini across every question type, with the Information Extraction gap driven by partial-match acceptances.

F Prompt Templates … G.3 Coverage and Per-Answer Accuracy

The appendix specifies MEMLENS’s prompt-driven construction and evaluation pipeline, then reports subset validity and coverage–accuracy trade-offs. Its results show that rankings transfer reliably, while refusal behavior substantially shapes apparent accuracy.

  • F Prompt Templates: MEMLENS uses canonical prompts for evaluation, haystack dialogue generation, question generation, evidence-session construction, and anti-shortcut filtering.Runtime placeholders are filled by the pipeline, while helper variants share the canonical-template skeleton.
  • F.1 Evaluation Prompts: The evaluation prompt supplies image-inlined conversation history and requires direct answers or “Insufficient information” when the context is insufficient.For answer-refusal items, the gold token is NOT_MENTIONED, while canonical refusal phrases are accepted by the judge.
  • F.2 Haystack Generation: The haystack generator combines persona, conversation summary, and recent life events to produce authentically multimodal dialogue turns.The prompt encourages conversationally relevant photo sharing and marks mentioned events with EVENT:ID tags.
  • F.3 Question Generation: Question-generation prompts return strict JSON with questions, answers, and supporting facts across the benchmark’s major question types.IE uses two-hop textual-to-visual alignment, TR encodes one timestamp through a clock image, and KU generates a four-fact preference-evolution chain.
  • F.3 Question Generation: MSR identity-match questions make an image the only signal resolving a vague reference to a named entity, while enforcing five anti-leakage rules.The rules prevent temporal, ownership, descriptive, textual, and cultural shortcuts.
  • G.2 Canonical 195-question Subset for Agent Evaluation: 195 canonical questions preserve benchmark rankings: Spearman ρ = 0.94 (p < 0.01, n = 6 direct LVLMs) at 32K between the subset and full benchmark.Agent accuracies reflect released pipelines and their documented input adaptations, including captions, composite images, or original images.
  • G.3 Coverage and Per-Answer Accuracy: 73.15% is GPT-5.4’s highest per-answer accuracy, but it refuses 36.05% of answerable questions; Qwen3.5-122B attempts 87.70% with 62.32% per-answer accuracy.Claude Sonnet 4.5 reaches 57.51% per-answer accuracy despite a 50.50% refusal rate and ranks 23rd overall.
  • G.3 Coverage and Per-Answer Accuracy: 63.55% and 62.96% are Claude Sonnet 4.5’s per-answer accuracies on IE and KU, despite overall accuracies of 28.05% and 29.31%.Its coverage ranges from 41.26% on MSR to 64.95% on TR, while refusal rates near 55% suppress its headline performance.

G.4 Wrong-Answer Error Analysis

The wrong-answer analysis uses a seven-label taxonomy that distinguishes near-miss errors, where evidence was located, from total-miss errors without a correct evidence anchor. Deterministic rules and LLM judgment identify these labels, revealing context-length shifts that aggregate categories would obscure.

  • Taxonomy: Four near-miss labels require located evidence, whereas three total-miss labels indicate content or non-answers without a correct evidence anchor.This split converts retrieval success into a binary signal for wrong-answer analysis.
  • Context-Length Shifts: 32K→128K aggregate shifts reach ±15.17% by partition identity, with unsupported answer alone carrying +10.23% while two other total-miss labels each move by less than 3%.The seven-way split is necessary to identify which labels absorb the shift.
  • Question-Type Signatures: Question-type signatures differ: MSR Arithmetic errors are computation slips, TR order-ranking errors are closed-set swaps, and KU errors are stale retrievals.Separating error labels from question types prevents distinct responses to context length from being merged.
  • Detection Procedure: Phase 1 assigns five labels using deterministic question-type and prediction-string rules, while Phase 2 uses GPT-4o-mini for the two remaining free-form labels.Phase 1 includes fuzzy matching for stale retrieval and refusal-keyword detection for answerability failure; Phase 1 labels take priority when both phases apply.
  • Coverage and Caveats: Phase 1 and the Phase 2 cache jointly cover roughly 90% of wrong answers per context, while residual invalid answers may inflate unsupported-answer share by at most ∼10% of its own value.The reported near-miss-to-total-miss shift remains stable under the default assignment.
  • Modality Regrouping: The complementary modality regrouping maps seven-label errors with image dependency into five disjoint categories: Visual, Textual, Mixed, Reasoning, and Output.Reasoning includes computation slip, closed-set selection, and answerability failure; Output is non-answer pathology.

G.5 Context-Length Analysis … G.8 Session Indistinguishability Validation

Across context-length, agent, MSR, and session-validation analyses, model scale and architecture shape retention, agents remain context-stable but lose information in retrieval and comprehension, and MSR is primarily retrieval-bounded. Construction and classifier tests further show that evidence sessions lack practically exploitable stylistic cues.

  • G.5 Context-Length Analysis: ρ = 0.62 links model size to stronger 32K-to-128K retention, while Qwen3-VL Instruct retention rises from 0.71 at 8B to 0.90 at 235B.Within Qwen3.5 dense models, retention is 0.77 at 27B, 0.65 at 9B, and 0.57 at 4B.
  • G.5 Context-Length Analysis: −13.18% versus −5.57% degradation distinguishes Qwen3.5-122B-A10B from Qwen3-VL-235B-A22B at 128K, showing family choice shapes both ceiling and degradation profile.Qwen3-VL preserves TR at 52–55% but loses KU to 30–24%, whereas Qwen3.5 preserves KU at 46.03% but lowers TR to 41.75%.
  • G.5 Context-Length Analysis: None of 72 nominal accuracy increases across 27 LVLMs and 7 agents survives Bonferroni correction, supporting monotonic degradation rather than a context-length artifact.Per-question churn explains apparent reversals: models lose 60–150 previously correct answers and gain 30–80 new correct answers at each transition.
  • G.5 Context-Length Analysis: −73.3% is the 32K-to-128K AR drop for Qwen3-VL 2B-Thinking, compared with −5.6% for the 235B Instruct tier, showing degradation depends on scale and decoding mode.The Thinking collapse reflects truncated reasoning traces producing substantive answers instead of refusals.
  • G.6 Agent Underperformance: Where in the Pipeline Is Information Lost?: 15.29% is Memory-T1’s advantage over M2A despite a 2.5× smaller text-only backbone, indicating pipeline architecture and task-specific training can outweigh backbone scale.M2A scores 14.21%, while Memory-T1 scores 29.50% and reaches 63.46% TR.
  • G.6 Agent Underperformance: Where in the Pipeline Is Information Lost?: 10.65% is the Mem0 gain from replacing Qwen3-8B with gpt-4.1-mini, whereas Qwen2.5-7B lowers accuracy by 4.00%, confirming backbone quality affects fixed memory pipelines.Within Mem0, AR ranges from 77.27% with Qwen3-8B to 100% with Qwen2.5-7B, while TR favors Qwen3-8B at 50.00% versus 32.69%.
  • G.7 MSR Ceiling Diagnostic: Retrieval-Bounded Difficulty: 100.00% for GPT-5.4 and 90.21% for Gemini-3.1-Pro on oracle-retrieval MSR show frontier models can aggregate three to eight evidence sessions once directly provided.The resulting 90–100% oracle-versus-full-benchmark gap identifies retrieval, rather than reasoning, as the MSR bottleneck.
  • G.8 Session Indistinguishability Validation: 54.77%–56.71% classifier accuracy, never more than 7 points above chance, remains after removing up to 1,000 discriminative n-grams, indicating no exploitable stylistic fingerprint.Evidence and haystack sessions share generation prompts, style constraints, and image integration; only embedded needle facts differ structurally.

H Detailed Limitations and Future Work

MEMLENS’s evidence and sessions are synthetic despite intensive human review, and its evaluation leaves concerns about generator overlap, judge validation, and offline-only temporal dynamics. The benchmark also addresses overfitting, leakage, and dual-use risks through safeguards and explicit usage guidance.

  • Limitations: MEMLENS uses LLM-generated evidence and haystack sessions, with ShareGPT and UltraChat dialogues as filler, while naturalness relies on intensive human-in-the-loop review.The review pipeline audits candidate questions and sessions against conversational-naturalness criteria.
  • Limitations: Gemini-3-Pro generates the final questions, while Gemini-3.1-Pro leads evaluation, leaving residual concern about intra-family generator familiarity rather than strict model-identity circularity.The construction pipeline constrains question content instead of rewarding a particular output style.
  • Limitations: 800 of 73,784 judge calls (≈1.08%) received cross-family validation, while 484 items underwent human verification; judge agreement was κ = 0.86 against consensus labels.Inter-annotator κ was not separately reported because disagreements were adjudicated to a single label.
  • Future Work: MEMLENS evaluates frozen histories offline at 32K–256K tokens, leaving online streaming evaluation of causality, irreversible updates, and forgetting to future work.A streaming protocol would preserve temporal causality between memory writes and queries.
  • Broader Impacts and Ethics: The benchmark anticipates leaderboard overfitting and leakage, using an anti-shortcut filter, canonical 195-question subset, frozen tags, and per-image perceptual hashes as safeguards.The authors also discourage using released dialogues for training or fine-tuning because exposing evaluation items would compromise diagnostic value.

I Design Rationale for the Memory Ability Taxonomy

MEMLENS’s five memory abilities are designed to jointly cover retrieval, reasoning, knowledge updating, and epistemic calibration in long-term multimodal conversations. The taxonomy distinguishes single-session extraction, cross-session reasoning, temporal awareness, current-state tracking, and evidence-based refusal.

  • Coverage and empirical distinctiveness: Together, the five abilities span retrieval, reasoning, and knowledge update with minimal redundancy across complementary single-session, cross-session, temporal, update, and refusal tests.IE and MSR differ by retrieval granularity; TR and MSR test complementary reasoning; KU and AR target opposite update-related failures.
  • Information Extraction: IE tests single-session retrieval by locating an evidence session and extracting facts, including entities visible only in images.The multimodal extension requires visual recognition before textual retrieval.
  • Multi-Session Reasoning: MSR tests cross-session aggregation by combining facts distributed across three to eight sessions through counting, arithmetic, and entity resolution.Questions require identifying multiple sessions, extracting pertinent facts, and combining them.
  • Temporal Reasoning: TR tests temporal awareness by combining natural-language dates, session timestamps, session order, and visual temporal cues.The temporal signals include utterances, metadata, sequence order, clock faces, and calendars.
  • Knowledge Update: KU tests current-state tracking across four successive updates, requiring agents to select the latest attribute value rather than superseded ones.This tests selective forgetting as well as recall when users revise preferences, statements, or plans.
  • Answer Refusal: AR tests epistemic calibration by removing all evidence and requiring agents to decline answering instead of hallucinating a plausible response.The ability targets a documented language-model failure mode and supports trustworthy behavior.
Loading 2605.14906v1…