PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Video Understanding: A Survey

Xinyu Deng, Siwen Luo, Daochang Liu

arXiv:2609.31713v2cs.CVcs.AI

TL;DR

Fixed video-language inference leaves systems without adaptive control over which evidence to inspect, retain, verify, and act upon. This survey formalizes video understanding agents and synthesizes their challenges, states, learning approaches, and evaluation. It identifies agentic-native temporal modeling and video-native agents as future directions.

  • Problem

    Video understanding research is moving beyond fixed inference, but the emerging field of agents that adaptively control video evidence lacks systematic survey coverage.

    Source: §I · p. 1

  • Method

    The survey formalizes video agents and organizes the literature by video-specific challenges, state-space paradigms, learning mechanisms, supervision, benchmarks, and evaluation.

    Source: §I · pp. 1–2, B. Video Understanding Agents · p. 2, §IV · p. 4

  • Results

    The synthesis links context bottlenecks to hierarchical memory, sparse evidence to active acquisition, temporal causality to state and process tracking, and multimodal ambiguity to role-specialized coordination.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The survey highlights agentic-native temporal modeling and video-native agent modeling, and calls for evaluation beyond final accuracy.

    Source: §IX · pp. 10–11

Abstract

from arXiv · show

As large language models (LLMs) become capable of processing increasingly diverse modalities and longer temporal contexts, an emerging line of work is moving beyond fixed video-language inference toward agentic systems that actively decide what information to inspect, retain, verify, and act upon. This survey reviews video understanding agents: systems that use video as the primary information source and solve understanding tasks through adaptive state construction and action selection. We first formalize an agent loop for video understanding, then address a central question: why do agents matter for video understanding? To answer this, we organize the literature through a challenge-to-design taxonomy, linking context bottlenecks to hierarchical evidence memory, evidence sparsity to active evidence acquisition, temporal causality to state and process tracking, and multimodal ambiguity to role-specialized coordination. We further review state space paradigms, learning paradigms, supervision signals, benchmarks, and evaluation protocols. Finally, we identify open directions toward agentic-native temporal modeling and video-native agents. Project page: https://github.com/DXY0711/Awesome-Agentic-Video-Understanding

I. INTRODUCTION

Video understanding has expanded from recognition to open-ended reasoning, but conventional Video LLMs generally follow fixed inference paths. This survey defines video agents by adaptive evidence control and organizes their design around video-specific challenges.

I. INTRODUCTION

  • Conventional Video LLMs typically receive evidence through a predetermined inference path, while difficult cases require deciding what to inspect, store, trust, revisit, and when to answer.Long videos may hide relevant brief events or distant states, and streaming systems may need to wait for future evidence.Source: §I · p. 1
  • The survey addresses a gap in prior surveys by organizing video agents around the interaction between video-specific challenges and agentic control.It links context bottlenecks to hierarchical evidence memory, sparse evidence to active acquisition, temporal causality to state and process tracking, and multimodal ambiguity to role-specialized coordination.Source: §I · pp. 1–2
  • The survey also reviews state-space paradigms, learning mechanisms, supervision signals, benchmarks, evaluations, and future directions.Its state-space view spans bags and sequences of frames, entity graphs, and evolving world states.Source: §I · p. 2

A. Video Understanding Tasks

  • The survey defines video understanding as reasoning over video-primary evidence, including visual content, temporal structure, multimodal cues, and evolving state.Its scope includes question answering, captioning, temporal grounding, retrieval, summarization, and streaming understanding.Source: A. Video Understanding Tasks · p. 2

B. Video Understanding Agents

  • A video understanding agent uses video as its primary source and solves tasks through adaptive evidence-state construction and action selection.Actions can change evidence access, state updates, verification, or stopping; fixed-sampling, one-pass Video LLMs are excluded.Source: B. Video Understanding Agents · p. 2

C. Survey Corpus and Inclusion Criteria

The survey corpus centers on recent agentic video-understanding work while including earlier material when needed for task or evaluation context. Inclusion requires video-primary input, a video-understanding task, and agentic control.

C. Survey Corpus and Inclusion Criteria

  • The core corpus draws mainly on recent work from major machine-learning and computer-vision venues, plus relevant recent preprints.It focuses on work released from 2024 to 2026, while admitting earlier foundational benchmarks, datasets, and methods when needed for context.Source: C. Survey Corpus and Inclusion Criteria · p. 2
  • Core-corpus inclusion requires video to be a primary input, the task to fall within the survey's video-understanding scope, and the method to exhibit agentic control.The resulting coverage includes offline and online, short and long, single- and multi-video, interactive, and streaming settings.Source: C. Survey Corpus and Inclusion Criteria · p. 2
  • Embodied or egocentric agents, computer-use agents, and video-generation or editing agents are considered only when they clarify relevant mechanisms, not as primary survey targets.The cited adjacent areas include video demonstrations, screen recordings, and video generation or editing.Source: C. Survey Corpus and Inclusion Criteria · p. 2

III. CHALLENGE-TO-DESIGN TAXONOMY

The taxonomy organizes video agents by the video-specific bottlenecks their designs address, rather than by mechanisms alone. For the context bottleneck, it links hierarchical evidence memory to preserving useful evidence across long videos and constrained context budgets.

III. CHALLENGE-TO-DESIGN TAXONOMY

  • The taxonomy assigns methods by the video-specific challenge their central contribution targets, since one method may address multiple bottlenecks.It connects memory to long-video evidence retention, search to sparse evidence acquisition, state tracking to temporal change, and collaboration to heterogeneous modalities.Source: §III · p. 2

A. Context Bottleneck: Why Video Agents Need Hierarchical Memory

  • Finite context and computation budgets encourage lossy sampling or compression, which can discard fine details and evidence across temporal scales.A single flat or fixed-scale context may struggle to preserve both global structure and fine-grained evidence.Source: A. Context Bottleneck: Why Video… · p. 3
  • Hierarchical evidence memory makes video evidence addressable at multiple temporal and semantic levels, with systems combining episodic, semantic, visual, and entity-linked memories.Examples include WorldMM, VideoARM, HAVEN [28], and AVI [30].Source: A. Context Bottleneck: Why Video… · p. 3
  • Compressed memory should retain links to its underlying frames or intervals, modalities, and entities to support later reasoning, retrieval, or decisions.Abstraction can introduce factual or temporal distortions that propagate; memory must balance compactness with enough grounding to recover or verify evidence.Source: A. Context Bottleneck: Why Video… · p. 3
  • Under streaming and long-horizon constraints, memory systems use caches or compression to avoid repeatedly processing full visual histories.Flash-VStream, StreamChat [32], and ProVideLLM [33] use cache or memory structures; StreamMeCo studies compression without losing retrieval quality.Source: A. Context Bottleneck: Why Video… · p. 3

B. Evidence Sparsity: Why Video Agents Need Active Evidence Acquisition

Video agents adapt evidence acquisition to sparse, query-relevant events, track how entities and events change over time, and coordinate specialized roles to reconcile heterogeneous modalities. These design dimensions address different challenges: what evidence to inspect, how to link it across time, and how to handle ambiguity across sources.

B. Evidence Sparsity: Why Video Agents Need Active Evidence Acquisition

  • Because relevant evidence may occupy little of a long video, agents iteratively inspect partial evidence, identify what is missing, and choose further observations or tool actions.Controllers plan retrieval, use curiosity or uncertainty to guide it, and may select tools, temporal scopes, or sampling densities; online systems may also decide whether to wait before responding.Source: B. Evidence Sparsity: Why Video Agents… · pp. 3–4

C. Temporal Causality: Why Video Agents Need State Tracking

  • State tracking links observations into evolving entities, relations, actions, and event chains when a single frame or segment cannot answer questions about change or temporal dependence.It may use storylines or entity-relation graphs; in online settings, it must update dependencies as evidence arrives and distinguish observed from unavailable information.Source: C. Temporal Causality: Why Video… · p. 4

D. Multimodal Ambiguity: Why Video Agents Need Role-Specialized Coordination

  • Role-specialized coordination separates and reconciles heterogeneous video evidence, addressing modality imbalance and conflicts among visual, speech, audio, and text cues.Coordination can use agents or dedicated modules; evaluation should consider evidence coverage, cross-modal correction, and groundedness beyond final accuracy.Source: D. Multimodal Ambiguity: Why Video… · p. 4
  • Module-specialized systems separate perception, memory, and reasoning or route signals through video-audio tasks and state-machine scheduling.OmniLive [102], MAViD, and ViSpeak illustrate different modular designs; G2F-RAG addresses textual-versus-visual evidence competition, while MAGNET integrates sparse audio-visual evidence across videos.Source: D. Multimodal Ambiguity: Why Video… · p. 4

IV. PARADIGM: STATE SPACE

The survey groups video-agent states into four paradigms: bags of frames, sequences of frames, graphs of entities, and evolving world states. The bag-of-frames paradigm treats selected visual units primarily as evidence, rather than as an explicit model of event evolution.

IV. PARADIGM: STATE SPACE

  • The four state-space paradigms differ in whether video evidence is organized as discrete units, temporally related observations, persistent relations, or causally updated states.The survey presents these as operative video states on which agents reason and act.Source: §IV · p. 4

A. Paradigm I: Video as a Bag of Frames

  • A bag-of-frames state comprises selected frames, keyframes, clips, shots, or segments that serve as evidence for recognition, grounding, or answers.EVA adaptively acquires observations; Refer-Agent [74] selects keyframes for object grounding, while A4VL and LVAgent retrieve candidate blocks or chunks.Source: A. Paradigm I: Video as a Bag of Frames · p. 5
  • Without explicit temporal order, entity persistence, or state transitions, bag-of-frames methods are limited for reasoning about event evolution and changes in the video world.They offer a compact abstraction compatible with existing VLMs and directly grounded in visual evidence.Source: A. Paradigm I: Video as a Bag of Frames · p. 5

B. Paradigm II: Video as a Sequence of Frames

Video-agent state representations range from ordered observations to persistent entity relations and causally updated world states. These paradigms differ in how they organize evidence over time and whether they maintain changing state as new observations arrive.

B. Paradigm II: Video as a Sequence of Frames

  • Sequence-based methods preserve the positions and relations of observations along a timeline, so segments are interpreted within an unfolding process.Examples include hierarchical segments in VCA, multi-scale observation paths in LensWalk, ordered GUI transitions in Mobile-Agent-V [25], and spatiotemporal trajectories in RoomTour3D.Source: B. Paradigm II: Video as a Sequence of… · p. 5

C. Paradigm III: Video as a Graph of Entities

  • Entity-graph states make associations across frames, segments, modalities, or reasoning steps explicit, supporting identity, temporal relations, and evidential links.Examples include GraphVideoAgent, HAVEN [28], and WorldMM; omitted details, incorrect links, or over-compressed memory can leave reasoning factually misgrounded.Source: C. Paradigm III: Video as a Graph of… · pp. 5–6

D. Paradigm IV: Video as an Evolving World State

  • Evolving-world-state methods maintain partial, time-indexed interpretations that can be revised as observations arrive, including uncertainty, missing evidence, and response readiness.Streaming systems such as AViLA [15] and ThinkStream [16] update state as evidence arrives; other systems track future-event anticipation, scene changes, risk, or interleaved state and action [90],,,.Source: D. Paradigm IV: Video as an Evolving… · p. 6

V. LEARNING PARADIGMS FOR VIDEO AGENTS

The survey groups video-agent behavior by how it is specified or learned, including inference-time control, supervised imitation, and reinforcement learning; these categories can be combined in multi-stage systems. Training-free agents use prompts, tools, retrieval, memory, verification, and stopping rules to control evidence gathering at inference time.

A. Training-Free and Inference-Time Control

  • Training-free agents assemble behavior at inference time rather than optimizing a dedicated agent policy.Their control can use prompts, tool interfaces, retrieval procedures, memory rules, verification mechanisms, and stopping or waiting criteria.Source: A. Training-Free and Inference-Time… · p. 6
  • Prompted tool-use systems decompose queries, inspect temporal regions or memory, call video tools, update evidence, and synthesize answers,,,,,,, [30].Their behavior is specified through prompts, retrieval modules, external tools, and video states rather than learned from trajectories or rewards.Source: A. Training-Free and Inference-Time… · p. 6
  • Prompt-defined role decomposition assigns specialized tasks such as planning, grounding, subtitle reasoning, visual perception, and reflection across agents,,.The survey notes that this structure can support modularity and verification even when roles are not learned.Source: A. Training-Free and Inference-Time… · p. 6
  • Inference-time reliability controls retrieve missing evidence, adjust tool calls, verify candidate answers, backtrack through memory, or decide whether to answer, retrieve, or wait, [14],,,, [15].Examples include DrVideo, VideoAgent2, VideoHV-Agent, VideoLucy, SEASON, and AViLA.Source: A. Training-Free and Inference-Time… · pp. 6–7
  • Inference-time control can use strong foundation models without process-level labels, but policies may oversearch, stop early, repeat calls, trust noisy tools, or follow weakly grounded plans.These are the survey's stated practical advantage and policy-fragility risks for this paradigm.Source: A. Training-Free and Inference-Time… · p. 7

B. Supervised Fine-Tuning (SFT) and Imitation Learning

The survey distinguishes answer-level supervised fine-tuning from imitation that supervises demonstrated decisions or trajectories, and describes reinforcement learning as reward-based optimization of agent decisions. It reviews their uses and limits, including demonstration coverage and reward credit assignment.

B. Supervised Fine-Tuning (SFT) and Imitation Learning

  • Answer-level SFT learns responses from videos and instructions, while imitation learning supervises demonstrated decisions or trajectories.In current video agents, imitation is commonly implemented as SFT on teacher-generated traces, so the distinction mainly concerns the supervision target rather than the optimization algorithm.Source: B. Supervised Fine-Tuning (SFT) and… · p. 7
  • Trajectory imitation can initialize an agent before later preference or reinforcement learning, using demonstrations of planning, grounding, tool use, and stopping decisions [62],,.VideoExplorer trains its planner and temporal grounder on successful trajectories; LongVideo-R1 and SAGE use tool-use traces for initialization.Source: B. Supervised Fine-Tuning (SFT) and… · p. 7
  • Supervised methods can train selected roles or evidence-chain generation, including planner, grounder, and verifier roles, rather than complete trajectories,,.VideoMind uses planning, temporal-localization, and verification labels; AoTD transfers verified traces into a single Video-LLM, while VITED trains temporal evidence chains.Source: B. Supervised Fine-Tuning (SFT) and… · p. 8
  • Supervised learning offers dense guidance when labels or traces exist, but its coverage and quality constrain it and demonstrations do not ensure adaptation to unfamiliar states.The survey identifies reproducing successful patterns without adapting to a new strategy as a limitation.Source: B. Supervised Fine-Tuning (SFT) and… · p. 8

C. Reinforcement Learning

  • RL optimizes agent decisions with outcome- or process-level rewards for correctness, grounding, efficiency, timing, or reasoning validity, including evidence acquisition and tool-use choices.Surveyed applications include perception and evidence acquisition,,,, [39], multi-agent coordination,, and temporal reasoning or grounding,,,.Source: C. Reinforcement Learning · pp. 8–9
  • RL fits decisions that lack direct supervision, but sparse rewards create credit-assignment difficulty because success may reflect accidental shortcuts rather than a reliable policy.The survey identifies reward sparsity as a central challenge.Source: C. Reinforcement Learning · p. 9

VI. DATA AND SUPERVISION FOR VIDEO AGENTS

The survey treats data and supervision as more than video-question-answer pairs: examples can record inspected evidence, tool use, memory, grounding, and stopping or revision decisions. It separates trajectory and grounding supervision and describes their roles in training and auditing agent behavior.

VI. DATA AND SUPERVISION FOR VIDEO AGENTS

  • Richer training examples can record inspected evidence, tool calls, memory writes, temporal grounding, and why an agent stops, revises, or responds.The section distinguishes trajectory supervision, grounding supervision, and preference and reward supervision as data-related aspects.Source: §VI · p. 9

A. Trajectory Supervision

  • Trajectory supervision records observations, decisions, state changes, and responses, with richer traces capturing failed branches, revisions, stopping decisions, and cost.Such traces support imitation and auditing whether an agent followed a grounded decision path rather than merely producing a plausible answer.Source: A. Trajectory Supervision · p. 9
  • Video-agent traces should identify the specific frames, clips, transcript spans, audio segments, tracks, temporal segments, or memory entries that informed each step.The trace should also record how those observations changed the next action; examples include timestamped ReAct traces, temporal evidence chains, and annotated visual temporal canvases, [82],,.Source: A. Trajectory Supervision · p. 9

B. Grounding Supervision

  • Grounding supervision links claims, answers, states, and actions to evidence such as temporal intervals, regions, identities, tracks, audio, transcripts, OCR, or state changes.In interactive settings, evidence timing also matters: an answer can be invalid if it relies on future evidence unavailable at decision time [15],, [80].Source: B. Grounding Supervision · p. 9
  • Grounded systems use temporal or localized evidence to generate or verify answers, and the survey calls for supervision that records evidence against competing hypotheses as well as supporting evidence,, [74],,.The cited systems address temporal, spatial, query-specific, or audio-visual grounding; the survey also highlights asynchronous query-evidence settings.Source: B. Grounding Supervision · p. 9

C. Preference and Reward Supervision

Preference and reward supervision compares or scores agent behavior, while benchmarks span capability tests and evaluations of evidence access, timing, tools, and external actions. The survey notes that final-task success remains the dominant benchmark emphasis despite broader process dimensions.

C. Preference and Reward Supervision

  • Preference and reward supervision supplies comparative or scalar judgments about answers, evidence choices, reasoning traces, tool calls, or complete rollouts.It differs from trajectory supervision, which records process, and grounding supervision, which anchors claims to evidence.Source: C. Preference and Reward Supervision · p. 9
  • Examples span video-search preferences, difficulty-aware or temporal-aware rewards, incremental reasoning and watch-think-speak timing, and routing based on readiness, latency, memory pressure, or model selection [105],,, [16], [39].The survey identifies reward validity as a challenge: correctness-only rewards may miss search efficiency, evidence reliability, or spurious reasoning.Source: C. Preference and Reward Supervision · pp. 9–10

VII. BENCHMARKS AND EVALUATION

  • Capability-oriented benchmarks test skills such as temporal grounding, long-range integration, multimodal reasoning, and procedural understanding, while typically providing the complete video before prediction.Benchmarks range from short-video QA, localization, and procedure understanding to long-context and diagnostic reasoning,,,,,,,,,,.Source: §VII · p. 10
  • Agent-oriented benchmarks add causal video access and decisions about response timing, tool use, web execution, or external retrieval,,, [144],, [140].Across benchmarks, evidence horizon, access regime, and action scope vary, but most still emphasize final-task success rather than evidence-process quality, timing, cost, and reliability.Source: §VII · p. 10

VIII. OPEN CHALLENGES AND FUTURE DIRECTIONS

The survey identifies two future directions: explicit operable temporal states and video-native representations of visual dynamics. It also calls for evaluation beyond final accuracy.

A. Agentic-Native Temporal Modeling

  • A temporal-state interface should support grounding, reading, updating, and stopping, while evaluation tests belief preservation, evidence-driven updates, moment selection, and avoidance of premature decisions.The interface links temporally anchored evidence and uncertainty to decisions to observe, retrieve, verify, wait, respond, or act.Source: A. Agentic-Native Temporal Modeling · p. 10

B. Video-Native Agent Modeling

  • Video-native agents could expose motion, state transitions, scene evolution, and action-conditioned outcomes directly to reasoning, especially in procedural, manipulation, and embodied settings.The direction addresses a potential loss of fine-grained visual dynamics when reasoning occurs in a language-dominated space.Source: B. Video-Native Agent Modeling · p. 10

IX. CONCLUSION

  • The conclusion highlights operable temporal-state and video-native modeling, and recommends evaluating evidence use, response timing, tool decisions, cost, and interaction reliability alongside final accuracy.The two directions focus on temporal states and reasoning over visual dynamics.Source: §IX · pp. 10–11

Related papers

Loading 2609.31713v2…