Source-linked AI summary
M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks
Jie Huang, Ruixun Liu, Sirui Sun, Xinyi Yang, Yin Li, Yixin Zhu, Yiwu Zhong
TL;DR
Existing video benchmarks do not systematically evaluate what multi-modal models retain, how faithfully they preserve it, or how memory withstands interference. M^3Eval introduces cognitively grounded video tasks spanning four memory dimensions and finds consistent weaknesses, including poor parallel-stream disentanglement, human-divergent interference patterns, weaker temporal than spatial source grounding, and limited symbolic memory.
Problem
Existing multi-modal video benchmarks primarily assess perception and reasoning and lack a dedicated protocol for systematically probing memory.
Method
M^3Eval uses controlled, cognitively grounded video question-answering paradigms to evaluate concurrent-input retention, interference robustness, interleaved-event organization, and symbolic memory.
Results
Experiments across open-source and proprietary models reveal failures in parallel-stream disentanglement, human-divergent interference behavior, temporal source grounding, symbolic memory, and irrelevant-information filtering.
Takeaways & Limitations
M^3Eval provides a systematic resource for studying multi-modal memory and highlights attention, interference handling, temporal grounding, symbolic memory, and filtering as future directions.
Abstract
from arXiv · showhide
As multi-modal models advance towards long-form video understanding, memory emerges as a critical capability. Despite substantial efforts in developing video datasets and benchmarks, existing works primarily focus on perception and reasoning, without systematically evaluating memory: what models retain, how faithfully information is preserved, and how robust memory remains under interference. To address this gap, we introduce M$^3$Eval, the first comprehensive evaluation framework and benchmark for probing different memory dimensions in multi-modal models. Grounded in cognitive psychology, our design features carefully constructed tasks that isolate key aspects of memory. Leveraging M$^3$Eval, we conduct extensive experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors. We find that models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns differing substantially from those observed in human memory, ground memory sources more reliably in the spatial domain than the temporal domain, and demonstrate limited symbolic memory. Collectively, our benchmark provides a valuable resource for future research, while our findings highlight memory as a fundamental yet underexplored capability and offer insights for designing more effective memory mechanisms in multi-modal models. Our code and dataset are available at https://pku-value-lab.github.io/m3eval-homepage.
1 Introduction
Existing video benchmarks mostly assess perception and reasoning, leaving multi-modal memory insufficiently isolated and systematically measured. M^3Eval addresses this gap with cognitively grounded tasks and finds distinct weaknesses across memory dimensions.
- Motivation: Long-form video and reasoning benchmarks implicitly involve memory but do not isolate memory mechanisms or systematically distinguish capacity, fidelity, and robustness.They therefore provide only partial and indirect assessments of memory.
- Framework: M^3Eval evaluates memory through controlled video-based question-answering tasks inspired by cognitive psychology.Its four dimensions cover concurrent inputs, interference, interleaved events, and abstract attributes.
- Findings: Models fail to maintain independent representations when processing parallel video streams, possibly because attention becomes confused across concurrent visual inputs.This finding concerns divided-attention encoding.
- Findings: Human and model interference patterns differ: humans show stronger retroactive than proactive interference, whereas models show comparable levels across the two conditions.Repeating interfering segments can even improve model understanding of target segments.
- Findings: Models ground memory sources more reliably spatially than temporally and show substantially weaker symbolic memory than humans.They also struggle to filter irrelevant information from memory.
- Contribution: The benchmark systematically evaluates diverse multi-modal models and offers insights into current memory limitations and future memory-mechanism design.The contribution includes a cognitively grounded design that isolates memory mechanisms through orchestrated video tasks.
2 Related Work
Prior video benchmarks evaluate perception and reasoning, while memory is often implicit or tested with simple retrieval probes. M^3Eval instead targets multiple memory dimensions through cognitively grounded evaluation paradigms.
- Video benchmarks: Video understanding benchmarks commonly conflate memory with visual perception and reasoning rather than measuring memory explicitly.This limitation spans general, long-form, streaming, and cross-video settings.
- Existing memory probes: Synthetic needle-in-a-haystack tasks test retrieval over long contexts but use simple probes that make different memory dimensions difficult to assess.A recent reasoning-based effort also does not systematically evaluate multiple dimensions.
- Positioning: M^3Eval differs from prior work by leveraging existing video datasets while explicitly probing key memory dimensions through cognitively grounded paradigms.This design connects controlled memory evaluation with video understanding.
- Cognitive foundation: Cognitive psychology separates memory into divided attention, interference, organization, and symbolic representation processes.These processes motivate M^3Eval's four evaluation paradigms.
3 Memory Evaluation
M^3Eval unifies four cognitively grounded video paradigms to measure distinct memory failure modes under concurrent inputs, interference, temporal interleaving, and symbolic abstraction. The framework uses controlled tasks, targeted questions, and metrics to quantify these capabilities.
- Framework overview: The framework evaluates divided attention, memory interference, interleaved events, and N-Back within one coherent design.Each paradigm targets a distinct memory process across spatial or temporal dimensions.
- Divided Attention: Divided Attention presents two semantically similar videos synchronously in split-screen form, with optional position swaps to test source-location correspondence.Questions measure source identification, order understanding, and content retention.
- Memory Interference: Memory Interference concatenates two semantically similar videos in reversed orders to isolate proactive and retroactive interference against the same target.Accuracy and Intrusion Rate quantify target recall and selections originating from the competing video.
- Interleaved Events: Interleaved Events alternates ten ordered segments from each of two source videos, requiring models to disentangle sources and recover target-video order.False Memory Discrimination tests whether models reject plausible but nonexistent content.
- N-Back: N-Back presents video clips controlled by lag distance N and sequence length K, testing symbolic matching across scene or action attributes.Accuracy is reported for whether the final clip matches the clip N positions earlier.
- Dataset construction: The benchmark contains 2,403 questions over 451 videos spanning approximately 403 hours, with materials drawn from five public datasets.Question-answer pairs are automatically generated and manually reviewed.
4 Experiments and Results
Experiments across proprietary, openweight, agentic, and human systems expose weaknesses in parallel-stream attention, interference resistance, temporal organization, and symbolic memory. Models often diverge from human memory patterns, especially in source grounding and filtering irrelevant information.
- Divided Attention: Models struggle with divided attention across parallel videos, with most near chance and substantially below human performance.Gemini-3.1-Pro-Preview is the exception noted in the reported results.
- Divided Attention: Attention becomes diffused and disorganized in split-screen inputs, likely causing confusion between concurrent visual streams.The reported hypothesis is that models cannot selectively attend to the relevant stream.
- Memory Interference: Models show comparable proactive and retroactive interference, unlike humans, and high intrusion rates indicate susceptibility to competing videos.Repeating either the target or interfering video improves target understanding.
- Interleaved Events: Models organize temporally interleaved events poorly: humans outperform all models, and many models fall below the 25% random baseline on false-memory discrimination.Spatial source grounding generally exceeds temporal source grounding, with some models below random performance.
- N-Back: Models lag humans in symbolic memory and fail to filter irrelevant information, degrading as total symbols K increases but not as temporal gap N increases.Human accuracy declines with increasing N and decreases modestly with increasing K, whereas model accuracy can remain flat or improve with N.
- N-Back: Across experiments, models retain all past inputs similarly, contrasting with human working memory’s tendency to maintain recent items and forget earlier ones.The paper attributes this contrast to attention mechanisms and hypothesizes that forgetting mechanisms could help.
5 Conclusion
M^3Eval systematically measures multimodal memory through cognitively grounded video tasks and reveals consistent weaknesses across models. The findings motivate improvements in attention, interference handling, temporal grounding, and symbolic memory.
- Benchmark: M^3Eval is the first benchmark to systematically measure multimodal memory across different dimensions.Its tasks are grounded in cognitive psychology and move beyond conventional video-understanding benchmarks.
- Implications: The results point to refining attention for parallel streams, using repetition against interference, strengthening temporal grounding, and improving symbolic-memory filtering.The paper presents these as future directions for developing robust, structured, and human-aligned multimodal memory.
A.1 Question Count
The full M^3Eval benchmark contains 2,403 questions spanning non-N-Back memory tasks and N-Back evaluations. Its questions are assembled from public video datasets and controlled sequence instances.
- Non-N-Back Questions: 739 non-N-Back questions evaluate divided attention, memory interference, and interleaved events using 451 videos from six public datasets.The source-dataset distribution is detailed in Table 4.
- N-Back Questions: 1,664 N-Back questions are generated from 64 twelve-clip sequence instances covering action and scene attributes.Each instance provides 26 valid K×N combinations across memory loads and temporal gaps.
A.2 Video Duration
The non-N-Back tasks use source videos with a range of durations. Figure 12 summarizes the duration distribution for the 451 videos used in those tasks.
- Video Duration: Each N-Back clip is trimmed from the non-N-Back source videos.
- Video Duration: Figure 12 shows the duration histogram for 451 source videos used in the non-N-Back tasks.These videos support divided attention, memory interference, and interleaved events evaluations.
B Video Construction Details
M$^3$Eval constructs video materials from diverse public datasets and forms semantically similar video pairs before generating localized memory questions. Its non-N-Back pipeline covers several memory dimensions, while a separate N-Back pipeline targets symbolic attribute tracking.
- Video sourcing: Videos come from five public datasets spanning egocentric activities, web videos, TV episodes, cooking tutorials, and movies.The source datasets are HourVideo, Video-MME, LVBench, InfiniBench, and CrossVid.
- Video pairing: Video pairs are constructed using a semantic similarity-first principle within each dataset.Similarity is based on topics, scenes, or narrative structure.
- Task pipelines: The non-N-Back pipeline covers Divided Attention, Memory Interference, Interleaved Events, and source-memory judgment tasks.
- Task pipelines: The N-Back pipeline targets tracking abstract scene or action attributes over a video stream.
- Question construction: Question construction proceeds through video segmentation, hierarchical description extraction, model-based generation, and manual filtering and verification.Filtering removes controversial or composite scenarios and checks that labels are free from interference and ambiguity.
- Question construction: Videos are segmented into short localized units that serve as the basis for later description extraction and question generation.
C.1.2 Hierarchical Description Extraction
The benchmark builds hierarchical, structured video evidence and task-specific questions for multiple memory dimensions. It combines predefined caption fields, controlled prompts, temporal attribute annotation, and manual review to reduce ambiguity and ensure unambiguous probes.
- Hierarchical evidence: Each video segment is represented with structured evidence at local and global levels using a predefined six-key caption schema.The schema includes main storyline, spatial relation binding, short-term action state, tool or prop, fine visual attribute, and text or symbol.
- Task-specific question design: Storyline-reconstruction options are controlled so each contains the same anchor clauses while differing through local reversals at designated slots.The construction specifies single-slot changes for B and C and a two-slot change for D.
- Task-specific question design: The non-N-Back prompts generate questions for source attribution, interference, source memory, storyline reconstruction, and related failure modes.The prompts ground question design in segment summaries, global summaries, and pair summaries or fine visual evidence, depending on task.
- Quality control: All generated candidate questions undergo manual review for logical consistency, plausible but incorrect distractors, and wording ambiguity.Only questions passing this quality check enter the final benchmark.
- N-Back annotation: N-Back annotation assigns scene and action phrases sequentially while reusing exact labels from a running memory when attributes recur.New phrases are introduced only for genuinely novel attributes, with a concise Action | Scene output format.
- N-Back construction: Four video groups are selected by clip-level attribute similarity, and three clips from each group are randomly combined into N-Back test sets.Manual review verifies attribute accuracy and a single unambiguous correct answer for each probe.
D Experimental Details
Experiments use model-specific frame-sampling defaults with exceptions for repeated-trial and N-Back settings. Locally deployed models run on four NVIDIA A800 GPUs, while proprietary models are accessed through official APIs.
- Frame sampling: Gemini-3.1-Pro-Preview uses 0.5 FPS, whereas other models use 96 uniform frames by default.Repeated-trial experiments use 144 frames, and N-Back experiments use 8 frames per clip.
- Execution settings: Locally deployed models are evaluated on a server equipped with 4 NVIDIA A800 GPUs.Proprietary models are evaluated through their official APIs.
E Example Visualization
The examples visualize the benchmark’s task formats and answer structures across divided attention, interference, interleaved events, source memory, and N-Back. They show how distractors alter source, order, content, or false-memory status while the correct option preserves the target criterion.
- Divided Attention: Divided Attention examples test source identification, order understanding, and content retention using distractors that mix, reorder, or fabricate target-video information.The correct option remains faithful to the target narrative.
- Memory Interference: Memory Interference presents the target-video answer alongside two intrusions from the interfering video and one unrelated distractor.
- Interleaved Events: Interleaved Events examples test source identification, order understanding, and content retention under videos whose events are interleaved.Distractors mix source content, swap sequence, or replace details with plausible fabrications.
- False Memory Discrimination: False Memory Discrimination asks the model to reject a relevant-looking question whose content belongs to neither video.The correct answer indicates that the query does not belong to either video.
- Source Memory: Source Memory uses a split-screen format with frequent left/right swaps to test spatial source attribution.
- N-Back: N-Back asks whether the final clip matches the clip N positions earlier on scene and action attributes.The response is a Yes/No judgment for each attribute.