Source-linked AI summary
MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism
Cong Chen, Guo Gan, Kaixiang Ji, ZhaoYang Zhang, Zhen Yang, Guangming Yao, Hao Chen, Jingdong Chen, Yi Yuan, Chunhua Shen
TL;DR
Hours-long videos overwhelm current Vision-Language Models with token explosion and attention dilution. MemDreamer decouples perception from reasoning through hierarchical graph memory and agentic retrieval, achieving SOTA results across four benchmarks while narrowing the human-expert gap to 3.7 points.
Problem
Current Vision-Language Models struggle with hours-long videos because massive, redundant visual token streams exceed context limits and dilute attention needed for long-range reasoning.
Method
MemDreamer streams videos into a three-tier Hierarchical Graph Memory and uses tool-augmented agentic retrieval to navigate task-relevant information during reasoning.
Results
MemDreamer achieves SOTA performance across four long-video benchmarks, narrowing the gap with human experts to 3.7 points.
Takeaways & Limitations
The results show a strong positive correlation between VLM agentic reasoning capability and long-video understanding performance, suggesting agentic-capacity scaling as a research direction.
Abstract
from arXiv · showhide
Current Vision-Language Models struggle with hours-long videos because processing full-length visual sequences induces prohibitive token explosion and attention dilution. To overcome this, we introduce MemDreamer to decouple perception and reasoning, shifting long-video understanding into an agentic exploration process. As a plug-and-play framework, it incrementally streams videos to construct a Hierarchical Graph Memory, a top-down three-tier architecture for semantic abstraction, anchored by a foundational graph capturing spatiotemporal and causal relations. During inference, the reasoning model employs agentic tool-augmented retrieval, navigating hierarchies, searching nodes, and traversing logical edges via an Observation-Reason-Action loop. Experiments show MemDreamer achieves SOTA results across four mainstream benchmarks, narrowing the gap with human experts to only 3.7 points. It constrains the reasoning context window to merely 2% of full-context ingestion while delivering a 12.5 point absolute accuracy gain. Furthermore, statistical analysis uncovers a strong positive linear correlation between an VLM's performance on logic reasoning and long-video understanding benchmarks, establishing agentic capability scaling as a new paradigm for multimodal comprehension.
1 Introduction
MEMDREAMER addresses long-video understanding by decoupling streaming perception from agentic reasoning over a persistent hierarchical graph memory. It reports state-of-the-art results across four benchmarks, including a 12.5-point gain using only 2% of the context window, while linking reasoning ability to long-video performance.
- Limitations: Current VLMs couple visual perception and logical reasoning through ultra-long context windows and brute-force frame sampling, creating severe long-video bottlenecks.The coupled approach flattens videos into massive token streams; sampling a 2-hour video at 1 FPS generates over 1.6M tokens.
- Approach: MEMDREAMER decouples perception and reasoning by streaming videos into a persistent memory bank, then retrieving task-relevant cues with a separate reasoning model.This streaming-and-retrieval mechanism is designed to circumvent context limits while preserving incremental video processing.
- Approach: Its Hierarchical Graph Memory organizes video content coarse-to-fine across three tiers, with cross-tier topological edges and leaf-level entity and micro-event subgraphs.The graph captures attributive and causal relations among fine-grained entities and events rather than storing a flat sequence of moments.
- Approach: Agentic Retrieval uses navigation, search, and graph-traversal tools in an iterative Observation-Reason-Action loop to locate and connect relevant memory nodes.The mechanism supersedes full-context ingestion and similarity-based retrieval by supporting hierarchical traversal and logical-chain tracking.
- Results: 12.5 points of absolute accuracy gain over end-to-end coupled paradigms are achieved on LVBench using only 2% of the context window, while the human-expert gap narrows to 3.7 points.MEMDREAMER also achieves state-of-the-art performance across four representative long-video benchmarks.
- Implications: MEMDREAMER reports a positive correlation between long-video performance and reasoning, and finds hierarchical graphs and agentic tool-use outperform flat storage and naive retrieval strategies.These findings motivate agentic capability scaling and foundational design principles for future multimodal memory systems.
2 Related Work
Prior work addresses long-video understanding through visual-encoder optimization, larger context windows, memory systems, and retrieval. However, existing VLM memory designs remain flat, while conventional retrieval may miss logical relevance and cannot dynamically self-correct.
- Long Video Understanding: Long-video research optimizes visual encoders and scales native context windows to handle potentially infinite visual inputs.Gemini-2.5-Pro is cited as expanding context limits to 1M tokens.
- Multimodal Memory: LLM memory systems such as MemGPT and MemoryBank have succeeded, but adapting memory systems to VLMs remains challenging.Early video approaches use simple frame buffers, while later systems encapsulate semantics into discrete bins.
- Multimodal Memory: Flat video-memory designs lack both top-down taxonomy and topological edges.The passage contrasts these limitations with memory approaches that encode video semantics into discrete bins.
- Information-Seeking Approaches: Naive retrieval-augmented generation uses static dense embeddings, but semantic similarity does not guarantee logical relevance.Visually similar clips may lack causal connection to the query.
- Information-Seeking Approaches: Single-turn retrieval methods cannot dynamically self-correct during information seeking.The passage presents this limitation alongside the failure of static semantic similarity to capture causal connections.
3 Method
MemDreamer decouples long-video understanding into streaming hierarchical memory construction and tool-augmented agentic retrieval. Its memory combines coarse-to-fine semantic organization with fine-grained spatiotemporal and causal graph structure, which the reasoning model explores through an Observation-Reason-Action loop.
- Decoupled Framework: MemDreamer first streams video through a perception model to construct a purely textual Hierarchical Graph Memory, then uses a reasoning model with tools to explore it for each query.This separates persistent memory construction from subsequent tool-augmented retrieval.
- Hierarchical Graph Memory: The memory organizes video into three semantic tiers—Video Root, Super Events, and Macro Events—while expanding each Macro Event into a local subgraph.The local subgraph represents entities, micro-events, and their spatiotemporal and causal dependencies.
- Memory Construction: Semantic-boundary-driven streaming segmentation produces self-contained Macro Events and reuses each window’s final event as the next window’s starting point.This avoids arbitrary fixed-window truncation and bounds the perception model’s input duration by the maximum window horizon τ.
- Memory Construction: Each Macro Event subgraph includes entity and micro-event nodes connected by spatial-attribute, subject-object, and directed temporal-causal edges.The perception model extracts this structured subgraph together with a textual Macro Event summary.
- Agentic Retrieval: The reasoning model retrieves evidence through hierarchical navigation, semantic or temporal search, and local graph traversal within an Observation-Reason-Action loop.It selects tools using the query and execution history, then distills observations into task-relevant clues instead of concatenating raw outputs.
4 Experiments
Experiments show that MemDreamer achieves state-of-the-art long-video understanding while substantially reducing reasoning-context demands. Ablations attribute its gains to decoupled perception, hierarchical graph memory, and agentic retrieval, with performance also correlating with logical reasoning ability.
- SOTA Performance Across Diverse Benchmarks: 90.7 peak score on LVBench establishes MEMDREAMER as SOTA, improving 12.5 points over Gemini-3.1-Pro and narrowing the human-expert gap to 3.7 points.The framework is evaluated across four long-video benchmarks, including LVBench, LongVideoBench, Video-MME, and EgoSchema.
- Paradigm Shift to Pure Agentic Reasoning: 0.702 Pearson correlation under end-to-end processing is statistically insignificant at p=0.052, linking raw-video ingestion with weaker alignment between logical capacity and LVBench performance.The analysis evaluates eight models on AIME2025 and compares their reasoning scores with LVBench scores.
- Hierarchy and graph contribute complementary gains: 77.4 is the worst 1D Flat-Chunk baseline, while 1D Flat-Graph reaches 84.8, a 7.4-point gain from adding topological edges alone.The result indicates that temporal and causal graph connections preserve event continuity; the combined Hierarchical-Graph design reaches 90.7.
- Tool-augmented retrieval beats embedding lookup: 90.7 for Agentic Full Tools surpasses Vanilla Embedding Similarity by 20.2 points, showing that multi-step tool invocation improves retrieval over static matching.The ablation evaluates progressively richer memory-access mechanisms, including search, graph traversal, and hierarchical navigation.
- Search Breadth: top-k Sweep: k=10 is the best search breadth, while k=5 reaches 87.1 and k=15 reaches 87.2, indicating a non-monotonic accuracy relationship.Too few candidates reduce evidence coverage, whereas too many dilute the informative cues with loosely related neighbours.
5 Conclusion
MemDreamer combines Hierarchical Graph Memory with tool-augmented agentic retrieval to establish a long-video understanding paradigm that bypasses context limits and mitigates attention dilution. It achieves new SOTA across challenging benchmarks and identifies a strong positive correlation between long-video performance and agentic reasoning capabilities.
- 5 Conclusion: MemDreamer integrates Hierarchical Graph Memory with tool-augmented agentic retrieval for long-video understanding.The framework is presented as a new long-video understanding paradigm.
- 5 Conclusion: MemDreamer establishes new SOTA across multiple challenging benchmarks.
- 5 Conclusion: The framework bypasses context limits and mitigates attention dilution, offering a promising scaling direction for future multimodal understanding.
- 5 Conclusion: The study reveals a strong positive correlation between long-video performance and agentic reasoning capabilities.The authors describe this correlation as the first revealed to their knowledge.
Appendix · 6 Reproducibility
The appendix specifies MemDreamer’s inference defaults, baseline-evaluation procedures, token-accounting setup, and artifact licensing. These details cover closed- and open-source reproducibility configurations and benchmark usage compliance.
- 6 Reproducibility: 16K tokens and temperature 0.7 are the retrieval defaults across Gemini-3.1-Pro, Gemini-2.5-Pro, and Qwen3-VL.These settings apply at retrieval time across all listed backbones.
- 6 Reproducibility: Hour-scale videos exceed the multi-GB ceiling of direct closed-source API uploads, so each video is re-encoded before submission.Gemini-family models use FPS=1 re-encoding and Google Cloud Storage staging to lift the size restriction.
- 6 Reproducibility: Closed-source baseline numbers come from technical reports or prior benchmarks where reported, with remaining cells produced in-house.The passage distinguishes reported results from in-house evaluations.
- 6 Reproducibility: 240k max tokens and the official prompt template are used for Qwen3-VL under the lmms-eval framework.Other open-source models use the framework’s default frame-sampling policy.
- 6 Reproducibility: 8192 tokens and temperature 0.7 govern all in-house open-source reproduction runs.Open-source baselines are run with lmms-eval.
- 6 Reproducibility: Over 1.6M tokens are claimed for a 2-hour 720P video sampled at 1 FPS using Qwen3-VL tokenization and 2 × 2 spatial merging.The setup specifies 7200 frames, 1280 × 720 resolution, a patch size of 32, and compression of every 4 adjacent tokens into 1.
- 6 Reproducibility: All artifacts comply with the licenses of four open-source video-understanding benchmarks and additional open-source visualization sources.The appendix identifies the benchmarks and visualization cases as existing open-source resources.
7 Qualitative Case Studies
The case studies show MemDreamer answering long-video questions by navigating hierarchical textual memory, exposing causal chains and recovering from failed retrieval through query reformulation. In contrast, direct end-to-end baselines can observe relevant shots but may lack the downstream causal context needed for the answer.
- Japanese culinary travelogue: The agent retrieves a subgraph that makes the pour → boil → spread → mesh-tray drying chain explicit, yielding the correct answer: (A) Drying them.The trace localizes the question to super_08, identifies macro_0020, and pulls its subgraph in round 2.
- Japanese culinary travelogue: The end-to-end Gemini baselines see the pour shot at the correct timestamps but lack the downstream causal context needed to answer correctly.All settings use Gemini-3.1-Pro; MemDreamer retrieves only the textual hierarchical graph without accessing frames.
- Nigerian news broadcast: After an uninformative first retrieval, the agent logs the failure, drops the unlikely cowboy-hat detail, and reformulates the query instead of committing to a weak hit.Round 1 returns no node satisfying all four cues; round 2 retrieves a relevant node after removing the detail unlikely to appear in textual descriptions.
- Nigerian news broadcast: Round 3 retrieves macro_0010, whose entity description and OCR overlay reading RIVERS POLITICS yield the answer (A) River Politics.The successful retrieval follows the agent’s logged negative result and query reformulation.