MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

• MEMLENS fills a benchmark gap by requiring visual evidence while enabling length-controlled comparison of long-context LVLMs and memory-augmented agents; Table 1 summarizes the missing prior coverage.Existing long-context, text-only conversational, and multimodal benchmarks lack this combination of visual necessity and direct comparison.Source: §1 · p. 2, Table 1 · p. 2
• MEMLENS comprises 789 questions spanning information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal across multimodal multi-session conversations.Its pipeline distributes evidence among coherent sessions with distractor turns, enabling extensible chat lengths and cross-modal token counting.Source: §1 · p. 2
• Removing images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images, confirming that visual evidence is necessary.The image-ablation study directly tests the benchmark’s cross-modal requirement.Source: §1 · p. 2
• Long-context LVLMs achieve high short-context accuracy through direct visual grounding but degrade with longer conversations, whereas memory agents remain length-stable while losing visual fidelity under storage-time compression.Memory-oriented post-training can additionally weaken memory agents’ abstention behavior.Source: §1 · p. 2
• MEMLENS evaluates 27 LVLMs and 7 memory-augmented agents across 32K, 64K, 128K, and 256K tokens, with multi-session reasoning capping most systems below 30%.The benchmark covers five largely independent memory abilities, so strong information extraction does not predict multi-session reasoning.Source: §1 · p. 2
• Neither long-context LVLMs nor memory-augmented agents comes close to solving long-term memory, motivating architectures that combine long-context attention with structured multimodal retrieval.The proposed direction avoids scaling either component in isolation.Source: §1 · pp. 2–3

Memory-Augmented LLM Agents

• Memory-augmented agents span structured, hierarchical, graph-based, embedding-based, reinforcement-learning, and multimodal retrieval designs for storing and selecting information.Multimodal extensions include document retrieval, multimodal embeddings, dual-layer semantic memory, entity-centric audio-visual memory, sparse video memory, session retrieval, and intent-guided response retrieval.Source: §2 · p. 3
• Recent work suggests long-context attention and retrieval-augmented memory are complementary rather than competing approaches.This line of research examines their intersection in long-context LLMs and retrieval-augmented generation.Source: §2 · p. 3

Long-Context and Conversational Memory Benchmarks

• Existing multimodal and conversational benchmarks either focus on documents or videos instead of multi-session conversations, or include images without requiring visual perception, leaving multimodal conversational memory insufficiently stress-tested.MMLongBench and related benchmarks scale multimodal context over documents, retrieval needles, video, super-long documents, or multiimage inputs, while LongMemEval, PerLTQA, and MemoryAgentBench omit images; LoCoMo and Mem-Gallery allow most questions to be answered from text alone.Source: §2 · p. 3

3 The MEMLENS Benchmark

• MEMLENS comprises 789 multimodal questions at 32K, 64K, 128K, and 256K tokens, requiring cross-modal reasoning over interleaved text and images.The benchmark targets multimodal long-term conversational memory rather than text-only conversations and questions.Source: §3 · p. 3

3.1 Memory Abilities

• The benchmark evaluates information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal.These abilities cover visual fact retrieval, aggregation across three to eight sessions, temporal interpretation, successive attribute updates, and calibrated refusal when evidence is absent.Source: §3.1 · pp. 3–4

3.2 Data Curation

• The four-stage construction pipeline simulates multimodal sessions, generates image-dependent questions, embeds evidence facts in realistic sessions, and assembles timestamp-ordered histories.Entity abstraction removes answer-critical names from text, while haystack and filler sessions create four standardized context lengths using cross-modal token counting.Source: §3.2 · pp. 4–5

3.3 Quality Control

• Automated filters and three human-review rounds reduce an initial pool of 20k candidates to 789 questions requiring evidence-image use.Review checks image-criticality, natural evidence embedding, plausible dialogue, and haystack quality; the remaining questions therefore require the evidence image.Source: §3.3 · p. 6

3.4 Cross-modality Validation

• 65.7% of questions are image-essential and 14.7% image-supportive; removing evidence images reduces both frontier LVLMs below 2% overall.With full evidence, GPT-5.4 reaches 93.13% and Gemini-3.1-Pro 89.42% on 634 image-essential and image-supportive questions.Source: §3.4 · p. 6

4.2 Main Results

• At 128K, Gemini-3.1-Pro retains 51.99% accuracy with only a 2.11% drop, while several open-weight LVLMs lose more than 13%; memory agents trail LVLMs especially on IE, KU, and AR.At 32K, the top eight LVLMs fall within a 6.34% accuracy band, whereas memory agents occupy a narrower range and multimodal agents perform substantially below top text-only agents.Source: §4.2 · p. 7, Figure 2 · p. 7
• MSR is hardest, with a 44.06% ceiling at 32K, while AR reaches 97.78%; only Kimi-K2.5 and Gemini-3.1-Pro exceed 30% on MSR.IE requires visual grounding, KU requires tracking a four-fact update chain, and MSR-Arithmetic is the hardest subtype overall.Source: §4.2 · pp. 7–8

4.3 Analysis

• Memory agents lose visual fidelity because storage-time captions and embeddings discard fine-grained image cues, producing the largest agent–LVLM gaps on IE and KU.The same bottleneck appears when text-only agents receive BLIP-2 captions, indicating lossy cross-modal storage rather than the answering model is the limiting factor.Source: §4.1 · p. 6, §4.3 · p. 8
• Fine-tuning memory-agent backbones collapses AR to 9–22%, including M2A at 22.73% versus 81.82% under direct inference on the same Qwen3-VL-8B backbone.Frozen-backbone Mem0 and MemOS retain 77.27% and 68.18% AR, suggesting current training rewards correctness and retrieval without sufficiently rewarding abstention.Source: §4.3 · pp. 8–9
• Six of seven memory agents stay within ±7% from 32K to 256K, whereas LVLMs lose approximately 20% on IE and 12% on KU as context grows.AR falls from approximately 75% to 45% for LVLMs, while MSR’s apparent flatness reflects a floor near 30%; the two approaches therefore have orthogonal failure modes.Source: §4.3 · p. 9
• Nearly 90% of IE and KU errors are visual, MSR errors are 73% reasoning, and oracle retrieval restores MSR accuracy to 90–100%, implicating upstream evidence retrieval.IE and KU form a retrieval-oriented correlation pair, whereas weak MSR correlations indicate a distinct cross-session aggregation difficulty; aggregate scores can therefore obscure ability-specific failures.Source: §4.3 · p. 9, Figure 5 · p. 9

5 Conclusion

• 58.68% is the strongest LVLM’s accuracy at 32K, while LVLMs degrade with longer contexts and memory agents remain length-stable but discard fine-grained visual evidence during storage.Post-training on memory-agent backbones can also erode abstention behavior.Source: §5 · p. 10
• MEMLENS is the first benchmark to evaluate LVLMs and memory-augmented agents under a unified, length-controlled protocol spanning 32K–256K tokens and five memory abilities.The benchmark targets multimodal conversational memory.Source: §5 · p. 10
• Nearly 90% of information-extraction errors arise from visual perception rather than comprehension, indicating that scaling first harms evidence retrieval rather than reasoning.Visual-evidence retention and retrieval emerge as the principal bottleneck for future progress.Source: §5 · p. 10
• Low cross-type correlation among the five memory abilities confirms that per-type evaluation is necessary.The benchmark’s abilities reveal distinct performance dimensions rather than a single transferable memory capability.Source: §5 · p. 10

Benchmark scope and release integrity

• MEMLENS is exclusively an evaluation benchmark, not training data, and frozen version tags make every leaderboard result traceable to its exact scored item set.The authors warn that supervised exposure to the 789 evaluation items would compromise diagnostic value.Source: Ethics Statement · p. 10

Data provenance and licensing

• The 4,695 source images exclude watermarked or copyright-overlaid candidates, avoid person-identification prompts, and include provenance metadata plus a seven-day takedown process.Author-produced materials use CC-BY-4.0, the evaluation harness uses MIT, and third-party images retain their original source-site licenses.Source: Ethics Statement · p. 10

Human review

• Project members, rather than crowd-workers, conduct review, with three sequential audits covering cross-modal necessity, session quality, and sampled haystack sessions.No third-party data collection or human-subjects protocol is involved.Source: Ethics Statement · p. 10

Broader applicability

• MEMLENS isolates five general memory abilities and may indicate robustness in document- or voice-grounded assistants involving retrieval, integration, and calibrated abstention.The authors frame these abilities as functional requirements of long-horizon conversational assistants beyond the benchmark’s multimodal setting.Source: Ethics Statement · p. 10

Reproducibility Statement

• Closed-source evaluations and synthetic-data construction use OpenAI, Anthropic, and Google APIs, with approximately 4,500 USD spent across pipeline construction and benchmarking.Open-weight LVLMs run locally with vLLM v0.17–0.18 on 8×A100-80GB nodes, using tensor parallelism for contexts at and above 128K.Source: Reproducibility Statement · pp. 10–11
• The 789-question benchmark, 4,695 provenance-linked source images, evaluation harness, and prompt templates are publicly released under frozen version tags for end-to-end reproduction.Per-model decoding, judging, retrieval depth, adapter details, construction routes, quality control, and agreement analyses are documented across the paper and appendices.Source: Reproducibility Statement · p. 11

B.1 Models

• The LVLM roster includes closed-source GPT-5.4 [2], Claude Sonnet 4.5, and Gemini-3.1-Pro, alongside open-source families including Qwen3-VL [76], Gemma3, and Qwen3.5.Table 4 specifies all 27 evaluated LVLMs, their training or claimed context lengths, image-processing methods, and context-extension techniques.Source: §B.1 · p. 18, Table 4 · p. 19
• Seven agents comprise three multimodal systems—M3-Agent [35], M2A, and M3C [37]—and four text-only systems: Mem0, MemOS [29], MemAgent-7B, and Memory-T1 [28].Their implementations combine retrieval, memory, backbone, and adaptation mechanisms described in the evaluation setup.Source: §B.1 · p. 18
• Released checkpoints are used for RL-finetuned or LoRA-adapted agents, while Mem0 [9] and MemOS [29] share Qwen3-8B [81] and receive alternative-backbone tests to separate architecture from backbone quality.The alternatives include gpt-4.1-mini for Mem0 and Qwen2.5-7B [82] for MemOS.Source: §B.1 · p. 18
• The evaluation covers 27 LVLMs at 32K–128K tokens and seven memory-augmented agents at 32K–256K, using a stratified 195-question subset for agents because their pipelines are substantially slower.M2A takes roughly 60× longer per question than direct LVLM inference.Source: §B.1 · p. 18
• Input adapters create a protocol asymmetry: text-only agents receive BLIP-2 [72] captions, M3-Agent receives composite session images, and only M2A and M3C process original interleaved messages directly.At answer time, text-only agents lack pixel access, M3-Agent re-attends a session composite, and M2A/M3C retrieve embedding-based memory entries.Source: §B.1 · p. 18
• Direct LVLMs attend to original pixels, whereas agent deficits conflate adapter-induced visual loss with retrieval and reading quality; matched-backbone and backbone ablations help isolate architecture.The study does not normalize adapters because released checkpoints assume their trained input formats; BLIP-2 captioning also bounds text-only agents’ visual-information ceiling.Source: §B.1 · p. 19, Table 5 · p. 20

B.2 Metrics

• 96.40% judge agreement with GPT-5.4-mini and 93.60% with human consensus support Qwen3-VL-235B-A22-Instruct as the canonical accuracy judge.Agreement was measured with Cohen’s κ = 0.93 and κ = 0.86, respectively; residual disagreement did not reorder the leaderboard.Source: §B.2 · p. 20
• LLM-as-Judge accuracy [73] is the primary metric because heterogeneous answer formats and verbose reasoning make substring matching unreliable.The judge reads the question, reference answer, and raw output, then emits a task-specific binary correctness verdict.Source: §B.2 · p. 20
• The evaluation pipeline tail-truncates outputs beyond 6,000 characters, auto-zeros responses exceeding 500 parsed words, and rejects circular reasoning.A format-dependent leniency for very short 1–3 word answers is documented and corrected in Appendix E.2.Source: §B.2 · p. 20
• Substring Exact Match checks whether the normalized reference appears in the output but can inflate scores when verbose reasoning traces mention the answer.The rule is generous to verbose models but is not the primary metric.Source: §B.2 · p. 20
• Coverage (Cov) measures attempts among 699 answerable questions, while Per-Answer Accuracy (PA) measures correctness on attempted answers, recovering overall accuracy through J ≈(Cov × PA × 699 + ARcorrect) / 789.This decomposition exposes the coverage–accuracy trade-off.Source: §B.2 · p. 20

B.3 Infrastructure

• Local models use vLLM v0.17–0.18 with FlashAttention-2 [89] on 8×A100-80GB nodes, using tensor parallelism for 128K inputs.API models use provider endpoints with 4–8 concurrent request threads.Source: §B.3 · p. 20
• Generation limits are 2,048 tokens for direct models and 16,384 tokens for thinking models to accommodate reasoning traces.These settings apply across the local and API serving infrastructure described for evaluation.Source: §B.3 · p. 20

C.1 Problem Formulation

• Each instance pairs a time-stamped multimodal session history with a query and answer, requiring evidence localization and cross-modal reasoning over answer-critical images.Answer-refusal items use the literal answer NOT_MENTIONED, while queries target five memory abilities.Source: §C.1 · pp. 20–21

C.2 Topic Ontology

• The ontology spans four answerable abilities, three tracks, approximately 100 topics per ability, approximately 30 subtopics per topic, and over 12,000 subtopics overall.The tracks cover identification, experience, and document content, organized into 19 sub-attributes.Source: §C.2 · p. 21

C.3 Subtype Detail

• MEMLENS includes nine subtypes with 789 questions, of which 65.7% are image-essential, 14.7% image-supportive, and 19.6% text-sufficient.Image-essential items are most prevalent in information extraction and multi-session reasoning, while temporal reasoning includes more text-sufficient temporal cues.Source: §C.3 · p. 22, Table 7 · p. 23

C.4 Question Construction Pipeline

• The construction pipeline generates backgrounds, retrieves visual anchors, masks entity names with under-specified anaphors, and verifies question-answer pairs against atomic evidence facts.Questions are constrained to require both image and surrounding text; multi-session reasoning additionally uses anti-leakage rules, rule-based filtering, and an LLM text-only judge.Source: §C.4 · pp. 22–25
• Subtype-specific routes isolate entity two-hop reasoning, prior-image recall, four-step knowledge updates, cross-session arithmetic/counting/entity reasoning, and temporal comparison or grounding.The routes include IE Entity (n = 120), IE PrevInfo (n = 126), KU Update (n = 116), MSR (n = 143), and TR (n = 194), with temporal modes covering visual and textual cues.Source: §C.4 · pp. 25–26

C.5 Image Filtering

• A unified image filter scores evidence and haystack candidates with CLIP, SigLIP, and BLIP-2-caption similarity, then removes negative content and near-duplicates.Candidates must exceed the CLIP threshold of 0.30 and one secondary channel; global URL uniqueness and pHash filtering prevent reused or near-duplicate images.Source: §C.5 · p. 26

C.6 Image Diversity

• Images are partitioned into natural photographs, text-rich documents, and digital or symbolic interfaces to cover distinct perceptual regimes rather than a single semantic axis.These formats support open-world recognition, OCR and layout grounding, interface parsing, synthetic clock/calendar reading, and visual evidence across IE, MSR, KU, and TR.Source: §C.6 · pp. 26–27

C.7 Image Sourcing, Licensing, and Release

• All images come from public web searches with uniform filtering, privacy safeguards, provenance metadata, licensing documentation, and reproducible release procedures.The release contains 4,695 unique images, while evidence sessions are randomly positioned; KU preserves the relative order of evidence sessions to maintain preference-update order.Source: §C.7 · pp. 27–28

D.1 Information Extraction

• Information-extraction questions abstract visually grounded entities or require recalling image details such as color, count, layout, or on-screen text from an earlier session.The latter questions therefore require multi-session image recall rather than text-only retrieval.Source: Figure 7 · p. 28, Figure 8 · p. 28

D.2 Multi-Session Reasoning

• Multi-session reasoning questions require arithmetic over scattered prices, durations, or quantities, counting matching sessions or items, or resolving whether cross-session references denote the same entity.At least one arithmetic operand is visible only in an image, while entity-resolution items support Yes/No identity matching or distinct-entity counting.Source: Figure 9 · p. 29, Figure 10 · p. 30, Figure 11 · p. 31

D.3 Temporal Reasoning

• Temporal-reasoning questions compare durations and establish chronological order or absolute dates using textual dates, session timestamps, and visual cues such as clock faces or calendar pages.The temporal evidence can therefore be distributed across modalities and sessions.Source: Figure 12 · p. 33, Figure 13 · p. 34

D.4 Knowledge Update

• Knowledge-update questions present a four-step preference chain anchored by a different image at each step, with the gold answer determined by the most recent state.The task tests whether the system updates knowledge as new multimodal evidence arrives.Source: Figure 14 · p. 35

D.5 Answer Refusal

• Answer-refusal questions deliberately remove supporting evidence from the conversation history, making a refusal phrase rather than a content answer the gold response.These items test whether the agent withholds unsupported answers.Source: Figure 15 · p. 36

E.1 Annotation Guidelines and Quality Assurance

• Question-level review checked image necessity, absence of answer leakage, answer unambiguity, and calibrated difficulty, while session-level review verified recoverable distributed needle facts, natural dialogue, and adjacent image–text references.Items needing revision were regenerated or rewritten, and items rejected by both reviewers were removed.Source: §E.1 · p. 36
• Naturalness was reinforced through human revision, real ShareGPT and UltraChat [65] filler conversations, and haystack audits that checked relevance, absence of spurious answers, image quality, and realistic dialogue.A text classifier achieved only DeBERTa F1: 57.92% when distinguishing evidence from haystack sessions, indicating little stylistic fingerprint.Source: §E.1 · pp. 36–37
• Three quality-control rounds reduced 20k candidates to 789 questions, with rejections driven by missing answer-critical image information (31%), difficulty failures (28%), answer ambiguity (18%), and question-text leakage (23%).Question-level review also found substantial accept/reject agreement, κ = 0.78 (Cohen’s κ; n = 200).Source: §E.1 · pp. 36–37

• 96.40% item-level agreement and Spearman ρ = 0.97 across judges show that judge choice does not reorder the leaderboard.The 800-item cross-family sample covers model families, context lengths, question types, and judge scores; the mean per-model accuracy delta is 3.70%.Source: §E.2 · p. 37
• 93.60% raw agreement with human consensus (Cohen’s κ = 0.86) confirms strong validity, despite 29 false positives versus 2 false negatives.The judge is leniency-biased, especially for partial-match information-extraction answers and hedge phrases in answer-refusal items.Source: §E.2 · pp. 37–38, Table 11 · p. 38
• The Qwen-family judge shows negligible family bias: the Qwen-versus-GPT leniency-gap difference is +0.33%, below the 3% practical-significance threshold.False-positive patterns depend on question type rather than output family, supporting judge-personality rather than family-specific bias.Source: §E.2 · pp. 37–38
• Format-dependent leniency affects very short answers, so false positives were corrected for 6 models and all reported scores use corrected values.The cross-judge gap is largest on information extraction (−6.50%) and zero on multi-session reasoning; no question type reverses ranking direction.Source: §E.2 · p. 38, Table 10 · p. 38
• 93.6% count-weighted agreement in a 12,234-item deterministic audit shows approximately 5% absolute LLM-judge inflation, but the leaderboard top remains unchanged.Agreement ranges from 87.4% on temporal order ranking to 98.5% on multi-session arithmetic; the deterministic and LLM metrics preserve the top five.Source: §E.2 · p. 39, Table 12 · p. 39

F Prompt Templates

• Canonical templates cover each MEMLENS pipeline stage, while runtime placeholders are substituted before calls and omitted helper variants remain available in the code release.The documented stages include evaluation, persona-driven haystack generation, question generation, evidence-session construction, and anti-shortcut filtering.Source: F Prompt Templates · pp. 39–40

F.1 Evaluation Prompts

• The evaluation prompt inlines multimodal conversation history, requires direct answers, and uses cached task-specific LLM judging for deterministic per-question accuracy.Reported numbers exclude chain-of-thought and structured-reasoning variants; the judge handles eight task-specific criteria.Source: §F.1 · p. 40
• Answer-refusal evaluation separates the stored gold token NOT_MENTIONED from the model phrase Insufficient information, with canonical refusal variants accepted by the judge.This distinction preserves a fixed dataset representation while allowing natural-language abstention at inference time.Source: §F.1 · p. 40
• The judge scores only a clearly committed final answer, assigning zero to inconsistent, purely unfinished, or refusal outputs except on answer-refusal tasks.The rubric uses a binary 1-point or 0-point scale and accepts semantic, numerical, synonym, and date-format equivalence.Source: §F.1 · pp. 40, 42

F.2 Haystack Generation

• Haystack generation combines persona profiles, prior-session summaries, and recent events to produce conversational user or assistant turns with naturally embedded image sharing.Event identifiers are appended at sentence ends, supporting traceable links between generated dialogue and life events.Source: §F.2 · p. 42

F.3 Question Generation

• Question-generation templates consume topic context, image metadata, and event tables, returning JSON questions, answers, and supporting facts before rule-based and judge filtering.Canonical prompts represent major question types, while subtype variants reuse their schemas with minor rule changes.Source: §F.3 · p. 43
• Information-extraction generation uses two-hop alignment: a textual fact identifies the visual target, and the answer must come from inspecting that element.The template varies visual skills across OCR, attributes, counting, and spatial relations while prohibiting explicit photo references.Source: §F.3 · p. 43
• Temporal order-ranking generation creates eight timestamped facts with one image-clock needle, requiring a later question date and modality-specific timestamp grounding.Related modes replace the clock with date or entity images while preserving the same generation skeleton.Source: §F.3 · p. 44
• Knowledge-update questions use four category-level preference facts ordered by time, with specific values supplied only through images.Each fact uses a distinct category and temporal marker, so text identifies the time step but not the preference value.Source: §F.3 · pp. 44–45
• Multi-session reasoning questions scatter named and vague entity references across time, making an attached image the only identity-resolution signal under five anti-leakage constraints.The constraints enforce temporal isolation, prevent ownership and descriptive leakage, prohibit text-based resolution, and preserve multiple plausible entity types.Source: §F.3 · pp. 45–46
• Answer-refusal questions remain topical but ask for information absent from the context, assigning the literal gold answer NOT_MENTIONED.The prompt requires an evidence snippet showing absence and a one-sentence abstention rationale.Source: §F.3 · p. 46

F.4 Evidence Session Construction

• Evidence sessions wrap each needle fact in an indistinguishable multi-turn conversation, targeting 250–350 words per assistant turn and validating generated turns through six leakage and format checks.The assistant template emphasizes helpful knowledge-oriented follow-ups while avoiding explicit signaling of the needle’s resolution role.Source: §F.4 · p. 47

F.5 Anti-Shortcut Quality Filter

• A text-only judge rejects questions answerable without images, while a complementary visual judge verifies that the image and question suffice to determine the answer.Together, the filters enforce the benchmark’s cross-modal grounding contract and block text-only shortcut solutions.Source: §F.5 · p. 48

G.2 Canonical 195-question Subset for Agent Evaluation

• The seven memory-augmented agents are evaluated on a 195-question canonical subset because their pipelines are substantially slower than direct-LVLM inference.M2A takes roughly 60× longer per question, motivating evaluation on one quarter of the 789-question benchmark.Source: §G.2 · p. 48
• The 195-question subset preserves each question type’s full-benchmark proportion within 0.2 percentage points, allowing rankings to transfer without systematic bias.All seven memory agents use this same stratified subset, and direct-LVLM runs are rescored on it for apples-to-apples comparison.Source: §G.2 · pp. 48–49, Table 15 · p. 49
• At 32K, rankings on the canonical 195-question subset transfer to the full benchmark with Spearman ρ = 0.94 (p < 0.01, n = 6 direct LVLMs).The subset therefore does not materially alter conclusions drawn from the full benchmark.Source: §G.2 · p. 50
• At 32K and 128K, text-only or caption-based agents cluster above multimodal pipelines, with agent confidence intervals spanning half-widths of 4.87–6.67 percentage points.The agent–LVLM gap remains larger than subset-induced uncertainty, preserving the qualitative cluster structure.Source: §G.2 · p. 50
• Under their released visual interfaces, current memory pipelines lose faithfulness to original visual evidence relative to direct-LVLM grounding while remaining robust from 32K to 256K.Text-only agents use BLIP-2 captions, M3-Agent uses composite session images, and M2A/M3C use original images; these results are not upper bounds for improved architectures.Source: §G.2 · p. 51

G.3 Coverage and Per-Answer Accuracy

• GPT-5.4 achieves the highest per-answer accuracy at 73.15% but refuses 36.05% of answerable questions, whereas Qwen3.5-122B attempts 87.70% at 62.32% per-answer accuracy.At 128K, most models increase coverage while accuracy falls, indicating false confidence rather than increased caution.Source: Table 17 · p. 51, §G.3 · p. 51
• Claude Sonnet 4.5’s coverage varies from 41.26% on MSR to 64.95% on TR, while MSR has low coverage and per-answer accuracy of 41.26% and 32.20%.Source: §G.3 · p. 51
• On IE and KU, Claude Sonnet 4.5 reaches 63.55% and 62.96% per-answer accuracy despite headline accuracies of 28.05% and 29.31%, respectively.Its roughly 55% refusal rate on these types suppresses overall accuracy, whereas timestamps provide explicit retrieval anchors for higher TR coverage.Source: §G.3 · p. 51

Taxonomy motivation

• The seven labels are necessary because unsupported answer absorbs +10.23% of the 32K→128K shift, while answerability failure and non-answer pathology each move by less than 3%.The seven-way split reveals which labels absorb context-length changes that meta-category totals conceal.Source: §G.4 · p. 52

Failure signatures

• Question-type signatures require label separation: MSR Arithmetic errors are computation slips, TR order-ranking errors are closed-set swaps, and KU errors are stale retrievals.Collapsing question type or error label merges failures with different responses to context length.Source: §G.4 · p. 52

Detection procedure

• Labels are assigned deterministically when possible, with five rule-based categories covering roughly 40% of incorrect attempts and GPT-4o-mini judging the remaining free-form cases.Phase 1 uses question type and prediction-string rules; cached Phase 2 classifications distinguish grounding failure from unsupported answer.Source: §G.4 · p. 52

Coverage and caveats

• The analysis covers roughly 90% of wrong answers per context; residual invalid answers may inflate unsupported-answer share by at most ∼10% of its own value.The default assignment leaves full-LLM relabeling for future work, while the near-miss-to-total-miss shift remains directionally and quantitatively stable.Source: §G.4 · p. 52

Meta-categories

• At 32K, near-miss errors comprise 69.44% of wrong answers, versus 30.57% for total-miss errors.Near-miss errors indicate evidence was located before the model erred; total-miss errors lack a correct evidence anchor.Source: §G.4 · p. 52, Figure 16 · p. 53

Modality regrouping

• The five modality categories combine each error label with image dependency, yielding visual, textual, mixed, reasoning, and output failures.Visual, textual, and mixed categories reflect image-essential, text-sufficient, and image-supportive questions; reasoning and output capture label-specific reasoning and non-answer errors.Source: §G.4 · pp. 53–54

Error shifts

• Unsupported answer rises 10.23% from 32K to 128K, replacing grounding failure (−5.38%) and computation slip (−4.82%).The shift is reported across wrong-answer types by question type, with n = 789.Source: Figure 17 · p. 54

G.5 Context-Length Analysis

• G.5 model size improves retention, with ρ = 0.62 across 22 open-source LVLMs; Qwen3-VL retention rises from 0.71 at 8B to 0.90 at 235B.Architecture also changes degradation profiles: Qwen3.5-122B-A10B loses −13.18% versus −5.57% for Qwen3-VL-235B-A22B at 128K, with different tradeoffs between TR and KU.Source: §G.5 · p. 56
• G.5 difficult questions remain unsolved despite longer context, averaging 9.07% accuracy at 32K and 8.52% at 128K.Answer-refusal degradation is also mode-dependent: Qwen3-VL-2B-Thinking falls from 87.8% to 14.4%, whereas the 235B Instruct tier drops only −5.6%.Source: §G.5 · pp. 56–57, Table 20 · p. 57
• G.5 validation finds expected context-length degradation overall: none of 72 nominal accuracy reversals survives Bonferroni correction, and apparent gains reflect churn, instability, or small samples.Across transitions, models lose and regain answers simultaneously; 38.7% of 32K→64K flips reverse at 64K→128K, while no systematic MSR positioning artifact appears.Source: §G.5 · pp. 56–57

G.6 Agent Underperformance: Where in the Pipeline Is Information Lost?

• G.6 agent errors split into retrieval and comprehension failures: M3C has 78.1% retrieval-attributed errors, whereas Mem0 and Memory-T1 have 87–95% post-retrieval errors.M3C’s mean evidence recall is 0.26, while Mem0 and Memory-T1 achieve recall of 0.82–0.89, motivating different interventions.Source: §G.6 · p. 58, Figure 19 · p. 59
• G.6 agent context stability does not offset information loss: six of seven agents stay within ±7% from 32K to 256K, yet their deficit versus LVLMs remains substantial.At 128K, context stability narrows the gap to Qwen3-VL-8B from 35% to 17%, while memorization and retrieval losses still dominate.Source: §G.6 · pp. 58–59

G.7 MSR Ceiling Diagnostic: Retrieval-Bounded Difficulty

• G.7 MSR diagnostics show that oracle retrieval raises MSR accuracy to 100.00% for GPT-5.4 and 90.21% for Gemini-3.1-Pro, identifying retrieval as the dominant bottleneck.Models can aggregate evidence once the required three to eight sessions are supplied, although Gemini-3.1-Pro retains an approximately 10% reasoning gap.Source: §G.7 · pp. 59–60

G.8 Session Indistinguishability Validation

• G.8 benchmark construction preserves evidence–haystack indistinguishability through shared generation, validation, chronological ordering, temporal consistency, and visual-density normalization.These constraints prevent evidence positions or surface style from providing shortcuts to retrieval.Source: §G.8 · pp. 60–61
• G.8 classifiers distinguish evidence from haystack sessions only marginally above chance, reaching 56.71% ± 1.20% and 57.92% ± 0.85% accuracy.The weak signal persists after removing up to 1,000 discriminative n-grams, indicating no practically exploitable stylistic fingerprint.Source: §G.8 · pp. 60–61

Synthetic conversations and human-in-the-loop naturalness

• Synthetic, human-reviewed sessions reduce stylistic shortcuts, but how well MEMLENS matches real long-term human–assistant interactions remains open.LLM-generated sessions undergo three rounds of human review; evidence-versus-haystack classifiers remain only marginally above chance (DeBERTa F1 57.92%, TF-IDF 56.71%).Source: H Detailed Limitations and Future Work · p. 61

Question generator and test-taker overlap

• Gemini-3-Pro generates questions while Gemini-3.1-Pro leads the leaderboard, creating a residual intra-family familiarity concern that targeted independent-generator re-ranking could test.Oracle retrieval diagnostics favor GPT-5.4 over Gemini-3.1-Pro, but no direct independent-generator ablation was conducted.Source: H Detailed Limitations and Future Work · p. 61

Judge limitations

• Judge validation is limited, with κ = 0.86 agreement, 29 false positives versus 2 false negatives, and unresolved inter-annotator and rare-subtype reliability.Only 800 of 73,784 judge calls were cross-family validated, and 484 items received human verification.Source: H Detailed Limitations and Future Work · p. 61

Static-length vs. streaming evaluation

• Evaluation is offline at frozen 32K–256K-token lengths, leaving streaming tests of causality, irreversible updates, and forgetting for future work.An online protocol would preserve temporal ordering between memory writes and queries.Source: H Detailed Limitations and Future Work · p. 62

Broader impacts

• MEMLENS exposes memory-faithfulness regressions before deployment, but leaderboard overfitting and synthetic-dialogue mismatch mean scores are necessary, not sufficient, indicators of deployed quality.Anti-shortcut filters, a canonical 195-question subset, versioned frozen tags, and source-site licensing support mitigation and reproducibility.Source: H Detailed Limitations and Future Work · p. 62

Ethics statement

• The benchmark should not be used for training or fine-tuning, because leakage could compromise its diagnostic value and faithful-memory regressions may amplify user-facing mis-recall.Frozen tags and per-image perceptual hashes support leakage detection, while licensing, privacy, and takedown procedures are documented.Source: H Detailed Limitations and Future Work · p. 62

Information Extraction (IE)

• IE requires locating evidence and extracting facts, including image-only entity recognition or recalling visual details from earlier sessions to test cross-modal memory.Entity-subtype questions require visual recognition before textual retrieval, while PrevInfo-subtype questions require recalling an earlier visual detail.Source: I Design Rationale for the Memory… · p. 62

Multi-Session Reasoning (MSR)

• MSR requires aggregating facts across three to eight sessions through counting, arithmetic, or entity resolution, with text-only items isolating reasoning from visual retrieval.The text-only subset also reflects that realistic conversations do not always encode cross-session information in images.Source: I Design Rationale for the Memory… · p. 62

Temporal Reasoning (TR)

• TR tests temporal inference from language, timestamps, session order, and visual temporal cues such as clock faces or calendar images.Its questions cover duration comparison, event ordering, and temporal grounding in long-running interaction histories.Source: I Design Rationale for the Memory… · p. 63

Knowledge Update (KU)

• KU presents four successive updates to one attribute and requires selecting the current value, testing both recall and selective forgetting of superseded information.It targets evolving preferences, corrections, and revised plans across sessions [57].Source: I Design Rationale for the Memory… · p. 63

Answer Refusal (AR)

• AR removes all evidence sessions from answerable instances, requiring the agent to refuse rather than hallucinate when supporting information is absent.This measures epistemic calibration and addresses a documented language-model failure mode [58].Source: I Design Rationale for the Memory… · p. 63

Coverage and empirical distinctiveness

• Cross-type correlations show that no ability pair is consistently strongly positively correlated across context lengths, supporting the taxonomy’s distinct coverage of retrieval, reasoning, and knowledge update.IE and MSR target single- versus cross-session retrieval; TR and MSR provide complementary reasoning tests; KU and AR probe revising versus abstaining.Source: I Design Rationale for the Memory… · p. 63