PaperlayerView exact revision on arXiv

Source-linked AI summary

DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

Boyuan Sun, Deshui Miao, Wenzhao Gao, Shaoyong Jia, Shaohui Jiao, Chi Liu, Yan Shu, Derek Li, Bryan Dai, Shengsheng Qian, Qibin Hou

arXiv:2610.19849v1cs.CVcs.AIcs.HC

TL;DR

Streaming systems must retain evidence before knowing what future questions will require, yet existing approaches can leave event changes implicit and omitted visual details unrecoverable. DeltaStreamer combines change-aware memory with query-driven planning and replay; it outperforms representative methods on OVO-Bench and StreamingBench, scoring 66.07 and 59.53 overall, respectively.

  • Problem

    Streaming systems must retain evidence before knowing which details future queries will need, while existing approaches can obscure event transitions and omit visual details that cannot later be recovered.

    Source: §1 · pp. 1–2

  • Method

    DeltaStreamer records inter-frame changes in traceable hierarchical memory, then uses query-driven planning, retrieval, and targeted replay to gather missing evidence.

    Source: §1 · p. 2

  • Results

    66.07 overall on OVO-Bench, +5.32 points over EventMemAgent; 59.53 overall on StreamingBench, +1.95 points over STRIDE.

    Source: §4.2.2 · p. 7

  • Takeaways & Limitations

    The findings support continuous observation and memory followed by active investigation when a question arrives.

    Source: §5 · p. 9

    The evaluation uses one shared backbone on two benchmarks, and precise proactive response timing remains challenging.

    Source: §D.1 · p. 22

Abstract

from arXiv · show

We propose DeltaStreamer, a novel agentic framework for streaming video understanding inspired by how humans observe, remember, and investigate. Streaming video understanding requires a system to retain evidence before knowing which details future queries will need. Existing approaches often obscure event transitions and omit relevant details when compressing video history, and cannot recover missing visual evidence. DeltaStreamer addresses these limitations by combining change-aware memory with query-driven planning and targeted evidence gathering with replay. Before a query arrives, it continuously observes the stream and explicitly records visual changes in structured memory, preserving how events unfold rather than only what is present. When a query arrives, a query-driven planner identifies the required evidence and guides memory retrieval and targeted replay. By revisiting historical frames, the agent can recover visual details omitted during observation and ground its answer in evidence beyond the retained descriptions. This connects continuous observation with active investigation tailored to each question. Experiments on OVO-Bench and StreamingBench demonstrate improvements over representative streaming video understanding methods, with overall scores of 66.07% and 59.53%, respectively. Project page: https://bbbbchan.github.io/DeltaStreamer/

1 Introduction

Streaming systems must retain evidence before knowing what future questions will require, but existing approaches can leave event changes implicit and cannot recover omitted visual details. DeltaStreamer combines explicit change recording with query-driven planning and replay to preserve and gather evidence for later answers.

1 Introduction

  • Because questions may arrive after relevant events, streaming systems must build and update memory before knowing which details will matter.Frames arrive sequentially, and future frames are inaccessible during current observation.Source: §1 · p. 1
  • Existing streaming methods can obscure event transitions through repeated representations and cannot recover visual details omitted after frames pass.They primarily capture what is present, while later queries may need evidence that was not retained.Source: §1 · p. 1
  • The framework treats streaming understanding as active information gathering: it observes and remembers continuously, then investigates when a query arrives.This approach is motivated by the possibility that stored representations may omit evidence needed by later queries.Source: §1 · p. 2
  • DeltaStreamer continuously builds hierarchical textual memory with source links and records inter-frame visual changes alongside scene descriptions.Its three-layer DAG, cross-layer pointers, and entity index make records traceable across temporal scales.Source: §1 · p. 2
  • A query-driven planner identifies evidence needs and guides memory retrieval, continued observation, and tool use; targeted replay can recover omitted pixel-level details.The persistent plan can direct the agent to historical frames through source links when descriptions are insufficient.Source: §1 · p. 2

2 Related Work

Prior streaming-video work supports retaining and accessing history through compression, memory, and organized representations, while agentic systems add planning and tool use. DeltaStreamer distinguishes itself by explicitly tracing semantic changes and keeping a query-specific plan available across retrieval and replay.

2 Related Work

  • Streaming-video approaches retain history through token compression, KV memory, dialogue, event memories, and retrieval of historical evidence.Examples include StreamForest’s event-level memory forest, WAT’s query-independent watching and query-triggered thinking, and Vista’s recall of full-resolution evidence.Source: §2 · p. 3
  • Agentic streaming systems extend memory retrieval with planning and tool use, including anticipatory planning, learned memory search, and coarse-to-fine retrieval.StreamAgent, EventMemAgent, and OASIS exemplify these directions.Source: §2 · p. 3
  • DeltaStreamer explicitly records adjacent-frame changes in traceable memory and maintains a structured query plan for later retrieval and targeted historical-frame replay.Its plan specifies the query target, temporal anchors, and required evidence.Source: §2 · p. 3

3 DeltaStreamer

DeltaStreamer observes streaming video before a query arrives, then plans evidence retrieval and targeted replay after the query to answer using accessible visual evidence. Its method combines delta-augmented observations, traceable hierarchical memory, query-driven planning, and look-back verification.

3.1 Problem Formulation and Overview

  • The system stores pre-query observations in a hierarchical graph, then plans retrieval, continued observation, or targeted replay after a query arrives.It can answer from available evidence or continue observing incoming frames until it can respond.Source: §3.1 · p. 3

3.2 Delta-Augmented Observation

  • Before queries, the VLM pairs each sampled frame’s scene description with changes from the preceding frame, while novelty scoring guides retrieval salience without deleting low-salience records.The change delta can describe an object appearing or changing state.Source: §3.2 · pp. 3–4

3.3 Traceable Hierarchical Memory

  • A traceable memory DAG links working observations, episodic summaries, and long-term pins; retrieval follows source pointers, while rendering retains summaries and recent records and keeps omitted records retrievable.The memory also includes an entity index mapping actor and object names to observation times.Source: §3.3 · p. 4

3.4 Query-Driven Planner

  • The planner identifies required evidence and specifies its target, temporal anchor, evidence type, temporal scope, and search strategy; its persistent evidence contract guides later retrieval and tool use.The agent may seek evidence in past, current, or incoming frames and adapt actions to updated memory and tool feedback.Source: §3.4 · p. 5

3.5 Look-Back Verification and Replay

  • A verifier checks retrieved records for cross-layer and temporal conflicts, then targeted replay inspects historical frames for missing details or unresolved conflicts before investigation resumes at the current frame.For verifier-triggered candidates, priorities favor severe conflicts and frames closer to question time; without an eligible candidate, no replay is requested.Source: §3.5 · p. 6

4 Experiments

DeltaStreamer is evaluated on two streaming benchmarks and ablations test its change-aware memory, query-driven planning, and evidence-gathering components.

4.1 Experimental Details

  • Evaluation uses Qwen3.6-27B at 1 fps on 8x NVIDIA A100 GPUs and covers OVO-Bench and StreamingBench’s real-time, recall, multimodal, and contextual tasks.OVO-Bench also tests proactive responses; inference uses vLLM in BF16 precision.Source: §4.1 · pp. 6–7

4.2.1 Compared Methods

  • Comparisons span proprietary models, offline video models, and streaming systems, with human results taken from the respective benchmarks.The listed streaming systems include StreamAgent and EventMemAgent.Source: §4.2.1 · p. 7

4.2.2 Benchmark Results

  • 66.07 on OVO-Bench exceeds EventMemAgent by +5.32 points, while 59.53 on StreamingBench exceeds STRIDE by +1.95 points.DeltaStreamer also outperforms Gemini 3 Flash by +2.05 points on OVO-Bench and StreamAgent on StreamingBench.Source: §4.2.2 · p. 7

4.3 Ablation Studies

  • With identical two-frame input, 2F-VisualDelta scores 66.07 versus 61.12 for 2F-NoDelta, supporting explicit recording of temporal changes.Delta fields make event transitions directly available for retrieval rather than requiring reconstruction from separate scene descriptions.Source: Table 3 · p. 7, §4.3 · p. 8
  • Hierarchical memory scores 66.07 versus 61.25 for flat memory, while DAG and entity-index ablations also favor retaining both access forms.Summaries provide an event-level overview, while source links and entity indexing help retrieve observations and timestamps; online episode construction also outperforms fixed grouping.Source: Table 4 · p. 7, §4.3 · p. 8
  • Planner additions reach 63.85 with plan generation and 64.77 with persistence; adding delta yields +0.93 without planning and +4.95 with planning.The larger conditional delta benefit is consistent with planning helping the agent use retained transitions.Source: §4.3 · p. 8, Table 5 · p. 8

4.4 Qualitative Analysis

  • Qualitative examples show recall of an action after the first block placement and recovery of missing wall letters from a historical crop.The examples illustrate how the planner connects query needs to observations and guides historical visual inspection when textual memory is insufficient.Source: Figure 4 · p. 9, §4.4 · p. 9

4.3 Ablation Studies

  • Adding middle recall raises accuracy from 63.69 to 66.07, while removing replay lowers it to 63.28, the largest drop among tool removals.Middle recall restores older stored observations; replay can recover visual evidence never encoded, while zoom and tracking address fine spatial details and cross-frame correspondence.Source: Table 6 · p. 8, §4.3 · p. 9

5 Conclusions

DeltaStreamer records inter-frame changes in hierarchical memory and uses query-driven planning to guide retrieval and replay when evidence is insufficient. Experiments and ablations support combining change-aware memory with planned evidence gathering; the supplementary materials document implementation, evaluation, and examples.

5 Conclusions

  • Explicit inter-frame changes in traceable hierarchical memory preserve event transitions for later queries.A query-driven planner specifies evidence needs and guides memory retrieval and targeted replay when evidence is insufficient.Source: §5 · p. 9
  • Experiments on OVO-Bench and StreamingBench, together with ablations, demonstrate benefits from combining change-aware memory with planned evidence gathering.The paper frames the findings as support for continuous observation and memory followed by active investigation when a question arrives.Source: §5 · p. 9

Appendix for DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

  • The supplementary materials document implementation settings, benchmark tasks and evaluation, memory and tool-use statistics, prompt templates, and qualitative examples.They also discuss limitations and broader impacts in the discussion section.Source: Appendix for DeltaStreamer: Towards… · p. 16

A.1 Observation Scoring Details

Observation novelty is scored from structured fields, then converted into retrieval salience and a change-bearing flag. Stored records link working events, episodes, and long-term pins while preserving source references.

A.1 Observation Scoring Details

  • The novelty score weights delta most heavily at 0.35, followed by actions at 0.25, actors at 0.15, objects and OCR at 0.10 each, and scene at 0.05.Description receives a default weight of 0.05; the history window contains up to 12 recent observations.Source: §A.1 · p. 17
  • Novelty of at least 0.3 yields salience 1 + 0.5s and a change-bearing flag; lower novelty receives salience 0.5.A repeated-delta guard demotes repeated templates to salience 0.5 and clears the change flag, which also informs episodic aggregation.Source: §A.1 · p. 17

A.2 Memory Construction and Rendering

  • Working events retain timestamps, text, deltas, salience, change flags, frame references, and optional episode links; episodes retain time spans, summaries, and supporting event identifiers.Long-term pins retain their source episode identifier, and reverse pointers support navigation without cycles in the hierarchy’s abstraction edges.Source: §A.2 · p. 17

A.2.1 Episode Construction

Episode construction combines online boundary detection with aggregation and coverage rules for observations not assigned to episodes. The appendix also specifies entity and memory retrieval, planner fields, and the evidence-gathering tool workflow.

A.2.1 Episode Construction

  • BOCPD proposes an episode boundary after at least three run observations when the most probable hypothesis indicates a new sequence, subject to a scene-and-actor continuity check.If the check does not suppress the boundary, accumulated observations are summarized into an episode and the detector resets.Source: §A.2.1 · p. 17
  • Uncovered observations are grouped chronologically using 10-second gap and 45-second span limits; retained groups need five events and a change-bearing observation.A coverage fallback creates lower-salience anchor episodes for sufficiently long static intervals, without requiring a detected change.Source: §A.2.1 · p. 17
  • The model can request an explicit episode through a Recap, which also creates a corresponding long-term pin when well formed.Episodes may also be created as observations arrive or when memory is prepared for a model call.Source: §A.2.1 · pp. 17–18

A.2.2 Entity Index and Memory Rendering

  • The entity index links rule-matched entities to working events, timestamps, and observed names, while Backtrack expands query-matching records into paths across memory layers.Visible memory ranks keyword-matching working records using salience, change, recency, and optionally token-hash similarity, then displays selected observations chronologically.Source: §A.2.2 · p. 18

A.3 Planner Schema and Evidence Guidance

  • The planner’s need and temporal anchor form the retrieval query, while evidence type, temporal locus, answerability, metadata, and strategy guide investigation and tool use.The strategy provides an initial course of action that the agent can adapt to new evidence.Source: §A.3 · p. 19

A.4 Evidence-Gathering Tools

  • Tools share a description and JSON-argument schema; the agent issues a tool call and waits for its result before continuing.The tool suite uses SAM2 for object tracking and Whisper for transcription; replay schedules historical-frame inspection and preserves the previous stream position for return.Source: §A.4 · p. 19
  • The tool suite is grouped by function.The table caption identifies the suite but does not specify its row or column encoding.Source: Table 7 · p. 19

A.5 Look-Back Verification and Replay Details

Look-back verification uses preliminary claim checks to decide whether conflicts warrant replay, then selects and ranks replay candidates. Replay adds frame-access and visual-processing overhead that can delay responses.

A.5 Look-Back Verification and Replay Details

  • Historical replay incurs substantial frame-access and visual-processing overhead, which can delay responses.The implementation assesses whether unresolved conflicts justify that cost before automatic replay.Source: §A.5 · p. 20
  • Groups with fewer than two claims do not trigger replay, while unanimous claims pass the agreement check and groups with at least 85% agreement are marked uncertain without replay.These preliminary outcomes concern replay need, not whether the proposed answer is visually correct.Source: §A.5 · p. 20
  • Working and episodic records with matching values pass the initial check despite differing long-term records; temporal gaps of at most two seconds pass the proximity check.Source: §A.5 · p. 20
  • Unresolved cross-layer conflicts and possible state transitions can nominate replay candidates, with severity reflecting the stronger signal and targets chosen from supporting working observations when available.The scheduler ranks candidates and discards priorities at or below 0.15, then deduplicates by target and caps replay at three.Source: §A.5 · p. 20

B Benchmark Details

The benchmarks evaluate streaming video understanding across real-time perception, recall, proactive response, multimodal understanding, and contextual reasoning. Their protocols also specify how query timing and response formats are evaluated.

B Benchmark Details

  • Query and response times use video timestamps, and queries beyond a supplied segment are mapped to its last available frame.Planning and finalization may reuse that frame without exposing additional video; recorded outputs and task-specific parsers determine accepted responses and formats.Source: B Benchmark Details · p. 20

B.1 OVO-Bench

  • OVO-Bench contains 3,035 samples across 12 tasks in real-time visual perception, backward tracing, and forward active responding.Tasks include recognition and prediction, episodic memory and hallucination detection, and repetition counting and clue-based responding.Source: §B.1 · pp. 20–21

B.2 StreamingBench

  • StreamingBench contains 4,500 QA samples across real-time visual, omni-source, and contextual-understanding groups.Its task coverage includes object and action perception, multimodal alignment, sequential question answering, and proactive output.Source: §B.2 · p. 21

C Memory and Tool Use

Memory grows quickly early in observation and then slows or stabilizes, while replay is concentrated during answering. The reported statistics describe recorded observations, retained memory, and initial plans.

C Memory and Tool Use

  • Visible memory grows quickly at first, then slows substantially on OVO-Bench and largely stabilizes on StreamingBench.Fig. 5(a) plots median rendered-input memory length by observation progress, with interquartile bands; it does not measure total storage of records and frames.Source: C Memory and Tool Use · p. 21, Figure 5 · p. 21
  • Replay is concentrated after query arrival, reflecting a trade-off between additional inspection and timely responses.Revisiting frames requires extra access and visual processing; tool budgets and termination rules help explain the low frequency of repeated replay.Source: C Memory and Tool Use · pp. 21–22, Figure 5 · p. 21
  • The reported statistics cover pre-query observations with both description and delta, stored records and episodes, and the first generated plan.Observation and delta text lengths are averaged in Unicode characters; working records include observations, frame references, and tool-related entries.Source: C Memory and Tool Use · p. 22, Table 8 · p. 22

D.1 Limitations

The authors identify latency, storage, error propagation, and evaluation scope as limitations, and note privacy and bias risks for potential deployments.

D.1 Limitations

  • Repeated model and tool calls may increase inference latency, while compact visible memory still leaves storage costs for records and historical frames.Frame arrival and processing can proceed asynchronously in real-world streaming settings.Source: §D.1 · p. 22
  • Errors in observations or temporal anchors can affect retrieval, and agreement across memory layers does not guarantee correctness.Source: §D.1 · p. 22
  • Evaluation uses one shared backbone on two benchmarks and leaves historical event localization, sequential-step recognition, and proactive response timing as challenges.The authors call for examining other backbones and trade-offs among accuracy, latency, and storage.Source: §D.1 · p. 22

D.2 Broader Impacts

  • Continuous observation, structured memory, and replay can expose personal routines and facilitate non-consensual monitoring.The authors recommend consent, limited retention, access controls, deletion mechanisms, and human review before consequential decisions; these safeguards were not evaluated.Source: §D.2 · p. 22

Related papers

Loading 2610.19849v1…