Source-linked AI summary
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
Jiahao Meng, Yue Tan, Qi Xu, Kuan Gao, Weisong Liu, Yanwei Li, Jason Li, Lingdong Kong, Haochen Wang, Qianyu Zhou, Jiangning Zhang, Guangliang Cheng, Yunhai Tong, Lu Qi, Minghsuan Yang
TL;DR
Long-video understanding requires models to select informative evidence, retain distributed context, and reason reliably across multimodal streams. This survey unifies these challenges through a human-view framework of watching, remembering, and reasoning, and organizes methods, datasets, benchmarks, applications, and open problems accordingly.
Problem
Long-video understanding lacks a unified account of how models select evidence, retain long-range context, and reason over distributed multimodal information.
Method
The survey models video understanding through watching, remembering, and reasoning, formalized as perceptual extraction, memory updates, and grounded inference.
Results
The survey organizes representative methods, challenges, datasets, benchmarks, and domain applications across the three functional abilities.
Takeaways & Limitations
The human-view taxonomy provides practical guidance for video model development, evaluation, and future research.
Takeaways & Limitations
Current medical video models still struggle with rare events, cross-domain transfer, and clinically faithful explanation.
Abstract
from arXiv · showhide
Video understanding is being rapidly transformed by multimodal large language models (MLLMs), as research moves from short clips to long, multimodal, and knowledge-intensive video scenarios. These scenarios require models to handle sparse evidence, long-range dependencies, multimodal alignment, and reliable inference under limited computational budgets. This work presents a human-view perspective on LLM-based video understanding, organized around three functional abilities: watching, remembering, and reasoning. Rather than treating video tasks as isolated benchmarks, this view provides a unified structure for analyzing how video MLLMs acquire evidence, preserve context, and produce grounded outputs. We introduce a formulation that characterizes video understanding systems by their perceptual representations, memory states, reasoning traces, and final predictions. Based on this formulation, we identify challenges in spatio-temporal perception, efficient long-video processing, memory modeling, streaming understanding, and faithful reasoning. Representative methods are organized by their roles in video MLLM systems. Watching covers fine-grained, comprehensive, audio-visual, and efficient perception. Remembering includes offline and streaming memory, while reasoning covers text-only reasoning and thinking with videos. We further examine application domains such as egocentric, sports, instructional, medical, and narrative videos, and cover training datasets and evaluation benchmarks across task types, supervision formats, modalities, and capability dimensions. Finally, we outline open problems and future directions for scalable, memory-aware, and evidence-grounded video intelligence. Related works will be continuously traced at https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding.
1 INTRODUCTION
The survey frames video understanding as a human-inspired Watch–Remember–Reason process for acquiring, retaining, and using multimodal evidence in long videos. It unifies recent advances, challenges, methods, datasets, benchmarks, and domain-specific applications within this taxonomy.
- Motivation: MLLM-based video understanding is shifting from short clips and isolated perception toward long-horizon comprehension spanning minutes to hours.This shift introduces challenges from redundant inputs and briefly appearing decisive evidence.
- Challenges: Long-video systems must selectively perceive informative moments, ground events across time and space, align modalities, and preserve salient information beyond finite context windows.The passage identifies compact memory, retrieval, and streaming mechanisms as responses to evidence sparsity and context limitations.
- Watch–Remember–Reason Taxonomy: The survey organizes video understanding around watching, remembering, and reasoning, corresponding to acquiring evidence, maintaining useful information, and deriving answers.The framework connects fine-grained grounding, long-video efficiency, memory augmentation, agentic understanding, streaming, and grounded reasoning.
- Resources and Applications: The survey also systematically summarizes training datasets, evaluation benchmarks, and domain-specific applications to guide model development, evaluation, and future research.This contribution extends the functional taxonomy from methods to practical resources and application areas.
- Survey Framework: Its unified formulation defines video inputs, outputs, memory states, and reasoning traces, then reviews representative methods through the three functional abilities.Watching includes temporal and spatial grounding, captioning, omni-modal perception, and efficient visual selection; remembering and reasoning cover memory and inference mechanisms.
2 BACKGROUND
This section frames video understanding as a unified, human-centered process of watching, remembering, and reasoning over long, complex multimodal streams. It formalizes these abilities as perception, memory updating, and inference components, then identifies challenges in reliable localization, detail preservation, and long-range evidence integration.
- Unified View: Video understanding is organized around three core abilities—watching, remembering, and reasoning—rather than treating tasks independently.The decomposition follows a human cognitive process and provides a unified perspective across diverse video understanding tasks.
- System Formulation: The system maps multimodal video and a query to outputs through watching, memory updating, and reasoning modules.Inputs may include visual frames, audio, and aligned text such as subtitles, ASR, or captions; outputs may include textual responses, temporal segments, or spatial regions.
- Functional Components: Watching extracts query-relevant perceptual evidence through operations such as spatiotemporal grounding, frame selection, cross-modal alignment, and semantic abstraction.Remembering accumulates useful evidence over time while filtering redundancy, and reasoning operates on perceptual evidence and memory to perform inference.
- Challenges: Key challenges include localizing continuous, overlapping, or sparse events; preserving fine-grained spatial details; and integrating distributed long-range evidence under constrained computation.These difficulties are compounded by occlusion, motion blur, viewpoint changes, temporal complexity, and cross-modal dependencies.
3 WATCH, REMEMBER, REASON: FROM FUNC- · TIONAL PERSPECTIVE
The section organizes video understanding systems around three functional abilities—watching, remembering, and reasoning—and reviews representative methods by their design principles, technical variations, and responses to associated challenges.
- TIONAL PERSPECTIVE: The review examines representative methods for watching, remembering, and reasoning, emphasizing their design principles, technical variations, and challenge-solving roles.These abilities form the section’s unified functional perspective on video understanding systems.
3.1 How to Watch?
Watching is the perceptual stage that converts raw multimodal video inputs into structured representations. It spans fine-grained grounding, comprehensive description, audio-visual alignment, and efficient evidence selection for scalable video understanding.
- Perceptual dimensions: Watching organizes video perception into fine-grained grounding, comprehensive semantic description, audio-visual integration, and efficient processing.Fine-grained watching localizes evidence in time and space; comprehensive watching summarizes videos and describes segments or regions; audio-visual watching aligns visual and acoustic streams; efficient watching reduces redundancy.
- Fine-grained watching: Temporal grounding localizes event intervals in untrimmed videos, increasingly treating timestamps as linguistic tokens within generative multimodal vocabularies.Recent methods address timestamp representation, long-video efficiency, structured decoding, fine-grained architectures, and verifiable post-training.
- Comprehensive watching: Comprehensive watching has expanded from concise clip captions to whole-video, dense, and region-level descriptions generated over visual-token sequences.Hierarchical methods connect local actions with global narratives, including recursive descriptions from clip and segment levels to full-video semantics.
- Comprehensive watching: Caption quality and controllability are improved through fine-grained supervision, preference or reward objectives, differential data construction, and user constraints on content, format, focus, or style.Examples include Tarsier2’s frame–event alignment and direct preference optimization, plus instruction-compliant and intent-oriented captioning.
- Efficient watching: Long-video methods select informative frames, compress redundant tokens, and optimize long-context computation to reduce overhead while preserving task-relevant evidence.These mechanisms support scaling video MLLMs to hour-long inputs under computational and memory constraints.
3.2 How to Remember
Memory systems for long-video understanding compress extended sequences into retrievable representations, balancing immediate grounding, persistent context, and limited context or memory budgets. Existing approaches organize memory through construction, storage, and retrieval, spanning agentic, non-agentic, and streaming paradigms.
- Memory Foundations: Memory compresses extended video sequences and retrieves relevant evidence, with short-term memory supporting current-context grounding and long-term memory persisting across tasks and sessions.Short-term memory includes KV caches and clip-level tokens, while long-term memory includes vector stores and textual summaries.
- Memory Foundations: Memory systems typically comprise construction, storage, and retrieval, using long-short-term, hierarchical, or event-graph storage structures.The approaches are categorized by how memory is constructed and utilized.
- Agentic Memory: Agentic methods invoke memory tools through multi-round LLM or VLM reasoning, enabling autonomous memory construction and retrieval but incurring substantial time and computational overhead.Agentic systems may use external memory and collaborative agents, whereas their multi-turn reasoning increases inference cost.
- Non-Agent Memory: Non-agent methods use deterministic sequential memory construction and retrieval, requiring only a single forward pass but additional training or cross-modal alignment.Their mechanisms have evolved from dense token retention toward sparse, structured consolidation, dynamic compression, and query-relevant selection.
- Streaming Memory: Streaming memory maintains historical context under fixed budgets through KV-cache compression, token reduction, recurrent selection, attention-sink reuse, and sliding windows.These mechanisms target unbounded video streams while keeping memory usage constant or preventing memory explosion.
- Streaming Memory: Streaming systems increasingly combine recent and long-term context with hierarchical or dual-memory designs and system-level CPU-GPU or streaming-KV co-design for real-time processing.Some frameworks also support proactive perception, decision, and reaction to dynamic visual stimuli.
3.3 How to Reason?
Video reasoning models combine semantic recognition, logical inference, and world knowledge to interpret perceived and remembered content. Existing approaches use explicit text-only thought traces, agentic video interaction, or training and optimization methods designed to improve temporal, spatial, and visual grounding.
- How to Reason?: Video reasoning models build on MLLMs to combine semantic recognition, logical inference, and world knowledge for deeper interpretations of objects, events, and relations.They commonly place an explicit <think>...</think> trace between markers before producing the final answer.
- Text-only Reasoning: Text-only reasoning decodes an intermediate language-space thought trace, including captioning, abstraction, step-by-step analysis, symbolic manipulation, and reflection.These methods broadly divide into agentic and non-agent approaches according to how the thinking process is organized.
- Agentic Approaches: Agentic approaches iteratively plan, retrieve, verify, revise, and search video evidence using modular tools, memory, adaptive attention, reflection, and answer verification.Examples include queryable spatio-temporal memory, tree-search exploration, multi-granularity browsing, segment retrieval, and frame inspection.
- Non-agent Approaches: Non-agent methods improve video reasoning through chain-of-thought supervision, temporal-aware or token-weighted GRPO, preference optimization, and video-specific rewards.Reward designs target temporal relationships, visual-semantic consistency, temporal IoU, task alignment, and factual event coverage.
- Challenges: Text-only reasoning can ignore visual cues, hallucinate statements, and produce long chains of thought whose grounding readers cannot easily verify.The section therefore emphasizes rechecking the video and strengthening spatio-temporal grounding during reasoning.
4 SUBFIELDS: VARIOUS VIDEO TYPES
This section reviews egocentric, sports, instructional, medical, and movie or narrative videos as distinct video-understanding settings with specialized evidence, temporal, knowledge, and reasoning demands. Recent work develops domain-specific benchmarks, multimodal models, memory mechanisms, and reasoning methods, while persistent challenges include reliable grounding, generalization, and faithful explanation.
- Egocentric videos: Egocentric videos require first-person interpretation of interactions, intentions, and 4D spatio-temporal dynamics, with recent work targeting fine-grained grounding, world modeling, navigation, and ultra-long reasoning.Ego-Mask provides pixel-level spatio-temporal grounding, while Ego-R1 coordinates hierarchical retrieval and tool usage for week-long video reasoning.
- Egocentric videos: Egocentric systems are also evolving toward proactive social interaction and safety-critical causal reasoning in streaming and driving scenarios.VideoLLM-EyeWO determines when to speak, EgoSocial addresses social intervention timing, and DVBench evaluates robustness in driving.
- Sports videos: Sports videos demand precise temporal grounding and sports knowledge because decisive evidence is time-localized amid fast actions, camera cuts, replays, and specialized rules.Recent approaches build domain-aligned datasets and curricula for rule-, tactic-, and soccer-specific reasoning, but models still struggle to localize decisive moments and apply rules consistently.
- Instructional videos: Instructional videos emphasize procedural progress, speech–vision alignment, and knowledge acquisition across long, information-dense lectures and tutorials.Video-MMMU and Video-MMLU measure learning-oriented understanding, while InstructionBench evaluates temporally ordered and procedurally structured reasoning.
- Medical videos: Medical videos require stable temporal modeling that combines global procedural context with local anatomical or tool-related evidence under strong domain and procedural constraints.The field is moving from specialized surgical recognition toward multimodal pretraining, medical VLMs and MLLMs, and evidence-based reasoning, yet rare events, cross-domain transfer, and clinically faithful explanation remain difficult.
- Movie and narrative videos: Movie and narrative videos require long-range understanding of scattered evidence, plot progression, character dynamics, causal links, and temporally grounded multi-step reasoning.Recent benchmarks evaluate event ordering, multi-turn decomposition, story understanding, series-level plot tracking, contextual movie QA, and deeper cognitive reasoning.
5 DATASETS AND BENCHMARKS
This section organizes video-MLLM datasets by task type, supervision format, modality, and video duration, covering Video QA, captioning, temporal grounding, and long-video memory. Across tasks, supervision progresses from conventional labels and large-scale instruction data toward grounded, reasoning-oriented, and tool-augmented training resources.
- Video QA: Video QA datasets progress from short-answer benchmarks to large-scale instruction corpora, then to one-shot CoT and agentic multi-round trajectories.VideoInstruct100K provides 100K instruction pairs, VideoChat2-IT scales to 1.9M samples, and later datasets supervise intermediate evidence, re-watching, tool interaction, and navigation.
- Video Captioning: Recent video datasets increasingly treat captions as general video-language supervision rather than only benchmark targets, supporting richer descriptions and text-to-video modeling.This shift follows automatic relabeling, recaptioning, hierarchical summaries, and specialized high-quality corpora.
- Video Temporal Grounding: Temporal-grounding datasets advance from web-scale pretraining and unified multi-task instruction tuning toward precision-oriented re-annotation and reasoning supervision.Representative resources include YT-Temporal-180M, TimeIT, VTG-IT-120K, Vid-Morp, VideoITG, TimeLens-100K, ActivityNet-RTL, TimeRFT, TVG-R1, VTTS-80K, and MTVR.
- Long Video Memory: Long-video memory datasets primarily support visual-linguistic alignment and memory-augmented understanding, extending the benchmark taxonomy beyond conventional task-specific supervision.The section identifies fine-tuning of alignment modules such as Q-Formers or cross-attention mechanisms as a central purpose.
V. Domain-Specific Knowledge
Domain-specific knowledge benchmarks evaluate whether video models can combine visual perception with specialized expertise across professional content. They distinguish expert-level reasoning from the ability to acquire knowledge through instructional videos.
- Domain-Specific Knowledge: Domain-specific benchmarks test the integration of visual perception with specialized expertise.The category covers professional content spanning scientific, medical, engineering, and humanities domains.
- Domain-Specific Knowledge: MMVU [334] emphasizes expert-level, knowledge-intensive reasoning over professional video content.Its evaluation focuses on applying specialized knowledge rather than merely recognizing visual information.
- Domain-Specific Knowledge: Video-MMMU [13] measures knowledge acquisition from instructional videos rather than recall of pre-trained knowledge.This gives the benchmark a complementary focus to MMVU’s expert-level reasoning evaluation.
6 FUTURE DIRECTIONS
Future work targets video understanding systems that reason about fine-grained spatial structure, multiple videos, long-range dependencies, and streaming interaction. Across these settings, the paper emphasizes stronger memory, efficient evidence search, verifiable outputs, and goal-driven online state updates.
- Spatial Reasoning: Spatial reasoning must improve from holistic scene description toward precise object localization, tracking, relationships, and coherent global scene-layout modeling.Promising approaches add dedicated visual encoders, explicit spatial representations for multi-view cues, and structured reasoning techniques.
- Multi-Video Grounding: Multi-video temporal grounding should retrieve candidate segments across videos and refine their boundaries, using edit-aware anchors and verifiable IoU-style rewards.Most existing methods assume one input video and therefore do not directly resolve cross-segment ambiguity.
- Long-Video Memory: Hour-long video understanding requires stronger memory to preserve rare decisive moments, track entities, and connect evidence separated by long temporal gaps.Compression, sparse selection, and periodic summaries reduce cost but can lose key details or break long-range dependencies; multi-level memory with evidence pointers is proposed.
- Efficient and Verifiable Reasoning: Long-video reasoning should become a budgeted evidence search that balances computational cost, answer correctness, evidence alignment, and evidence compactness.Efficient inspection, uncertainty-aware evidence requests, and standardized schemas for timestamps, boxes, and grounded captions can support verifiable outputs.
- Streaming Egocentric Understanding: Streaming egocentric video calls for stateful, goal-driven memory that updates online under latency constraints while maintaining coherent goals and environmental context.Future directions include explicit task state, event-triggered writing, proactive retrieval, and evaluations covering timing and stability under updates.
7 CONCLUSION
The survey frames MLLM-based video understanding through watching, remembering, and reasoning, highlighting progress across perception, memory, and reasoning. It identifies a shift toward structured memory, streaming systems, and explicit evidence-grounded reasoning.
- Conclusion: The survey organizes MLLM-based video understanding around watching, remembering, and reasoning.These functions provide the survey’s human-view perspective.
- Conclusion: It summarizes progress in spatio-temporal perception, efficient observation, memory construction and retrieval, and reasoning-centric training and evaluation.
- Conclusion: Recent work is shifting from simple input compression and answer generation toward structured memory, streaming systems, and explicit evidence-grounded reasoning.