Source-linked AI summary
Dynamic Hub-and-Spoke Memory for Streaming Video Understanding
Xinru Jiang, Lin Zhao, Xi Xiao, Yunbei Zhang, Janet Wang, Chenrui Ma, Haolin Li, Yanzhi Wang, Yifan Gong, Octavia Camps
TL;DR
Streaming video understanding needs to retain long-range visual evidence while retrieving question-relevant history from a continuously growing stream. D-HSM addresses this with training-free entity-centered textual memory and dynamic retrieval combined with recent visual frames, and it outperforms existing online and offline baselines across streaming and long-video benchmarks.
Problem
Streaming video understanding must compactly remember long-range history while efficiently retrieving evidence relevant to questions asked over a continuously growing visual stream.
Method
D-HSM converts selected historical chunks into structured textual observations, organizes them in entity-centered hub-and-spoke memory, and dynamically retrieves linked evidence alongside recent visual frames for frozen-VLM prediction.
Results
D-HSM achieves state-of-the-art performance on streaming benchmarks and retains strong performance on conventional offline long-video benchmarks, outperforming online and offline baselines.
Takeaways & Limitations
Structured, question-aware memory provides a way to compactly remember long video history and reconstruct relevant evidence on demand without additional training.
Takeaways & Limitations
Errors in frozen-VLM-generated observations, including entity recognition, OCR, or action descriptions, may propagate into memory, while textual storage can lose fine-grained distant details.
Abstract
from arXiv · showhide
Streaming video understanding requires answering questions at arbitrary times over a continuously growing visual stream. The central challenge is to compactly remember long-range history while effectively retrieving question-relevant evidence. We propose Dynamic Hub-and-Spoke Memory (D-HSM), a training-free framework that represents distant history as structured textual memory while preserving the recent frames as visual tokens for fine-grained perception. Specifically, D-HSM turns selected historical video chunks into typed textual observations and stores them in an entity-centered hub-and-spoke memory, with entities as hubs and related evidence as spokes. When answering a question, D-HSM dynamically retrieves a compact question-aware memory subset, expands it through hub-and-spoke links, and combines it with the recent visual window for frozen-VLM answer prediction. Extensive experiments on both streaming and long video benchmarks show that D-HSM consistently and substantially improves VLM backbones and outperforms other state-of-the-art online and offline video understanding baselines.
1 Introduction
Streaming video understanding must compactly preserve long-range visual history while retrieving only the evidence relevant to each question. D-HSM addresses this with structured entity-centered textual memory, adaptive retrieval, and recent visual frames, achieving strong benchmark performance without additional training.
- Streaming models must preserve long-range visual evidence despite the infeasibility of retaining the entire continuously growing stream in raw visual form.
- Existing methods reduce history through token selection, visual compression, bounded memory, or cached-representation retrieval, but leave history representation and organization open.
- Textual traces retain distant history under limited context budgets, but flat captions lack explicit links among recurring entities, actions, and relations across time.
- Real-time questions may be answered from recent frames, whereas backward-looking questions require selectively retrieved historical evidence, motivating adaptive retrieval.
- D-HSM converts selected historical chunks into structured observations, organizes recurring entities as hubs with evidence spokes, and keeps recent frames visual for fine-grained perception.
- D-HSM dynamically gates retrieval, selects a salient question-aware memory size instead of fixed top-K, expands linked evidence, and combines it with recent visual tokens for frozen-VLM prediction.
- D-HSM achieves state-of-the-art performance on streaming benchmarks and retains strong performance on offline long-video benchmarks without additional training.
2 Related Work
Prior long-video and streaming methods primarily extend, sample, compress, or cache visual representations, while graph-memory approaches are mainly evaluated offline. D-HSM instead maintains persistent entity-centric textual memory for incremental, question-adaptive retrieval.
- Long and Streaming Video Understanding: Offline long-video methods extend visual context by sampling more frames or scaling representations, with computational costs that grow rapidly with video length.
- Long and Streaming Video Understanding: Some methods organize events into graph memories and retrieve query-relevant subgraphs to reduce cost, but are primarily evaluated in offline settings.
- Long and Streaming Video Understanding: Online approaches process frames sequentially and manage history through sampling-based or storage-based visual compression.
- Long and Streaming Video Understanding: Sampling-based compression reduces input tokens but can discard fine-grained spatial or temporal details, while storage-based methods retain encoded visual memory.
- Long and Streaming Video Understanding: These methods largely retain history as features without explicitly organizing relationships among different entities.
- Long and Streaming Video Understanding: D-HSM maintains persistent entity-centric memory that is incrementally updated as chunks arrive, enabling compact and question-adaptive retrieval.
- Video Large Language Models: Video-LLMs map spatiotemporal visual features into language-model spaces to support open-ended dialogue and reasoning.
3 Method
D-HSM builds an online, entity-centered textual memory for distant history while retaining the recent visual window, then retrieves question-relevant evidence adaptively for a frozen VLM. Its memory uses typed observations, persistent entity hubs, timestamped spokes, and dynamic expansion to connect evidence across time.
- Framework: D-HSM preserves recent video as visual frames while converting longer-range history into compact textual memory for frozen-VLM inference.The framework maintains textual memory for historical chunks and combines retrieved evidence with the recent visual window.
- Hub-and-Spoke Memory: Entity hubs anchor persistent identities, while timestamped spokes store associated actions, relations, OCR text, and events across moments.Co-occurrence edges connect entities appearing in the same chunk, and occurrence counts and timestamps support temporal localization.
- Structured Textual Observations: Selected historical chunks are converted into typed observations containing objects, people, actions, OCR text, spatial relations, and events.Persistent identifiers allow recurring entities to be merged across chunks while complementary evidence remains in separate fields.
- Memory Update: D-HSM updates memory online by linking new entities to similar hubs, merging repeated spokes, adding novel facts, and removing unsupported evidence.Support tracking and inverted indices prevent stale evidence from remaining after historical chunks leave the active set.
- Dynamic Retrieval: Question-adaptive retrieval selects a compact subset using query-memory similarity and a salient score-gap cutoff instead of a fixed top-K size.The selected evidence is then expanded through hub-and-spoke links before being combined with recent visual tokens for answer prediction.
- Evaluation Setup: Table 1 compares D-HSM with proprietary, offline, and online models on StreamingBench using 20 historical frames plus 4 or 8 recent frames.The table evaluates Qwen2.5-VL and Qwen3-VL backbones under the 20+4 and 20+8 settings.
4 Experiments
D-HSM is evaluated across streaming, offline long-video, and component ablations, consistently improving performance while showing benefits from structured memory and adaptive retrieval.
- Streaming Video Understanding: D-HSM raises StreamingBench overall performance from 73.7 to 84.7 with Qwen2.5-VL and reaches 85.4 with Qwen3-VL.It surpasses Gemini 1.5 Pro and Streamforest by 9.7 and 8.1 points, respectively.
- Streaming Video Understanding: D-HSM achieves competitive performance across Real-Time Visual Perception, Backward Tracing, and Forward Active Responding on OVO-Bench.The reported pattern indicates adaptive use of recent visual and historical evidence across question types.
- Offline Long Video Understanding: D-HSM reaches 60.7 on LongVideoBench, 67.3 on MLVU, and 63.9 on VideoMME, outperforming the Qwen2.5-VL baseline and several open-source models.These results use Qwen2.5-VL with four recent frames.
- Ablation Study: Combining recent frames with HSM provides the best backward score without sacrificing real-time accuracy, while typed and entity-centered organization improves over flat chronological captions.Removing co-occurrence edges, next-action chains, or spoke merging consistently reduces performance.
- Ablation Study: Dynamic retrieval improves performance over fixed top-12 retrieval, while increasing the maximum budget beyond 12 or the historical budget beyond 20 can reduce performance.The budget results support balancing evidence coverage against redundant or weakly relevant context.
5 Analysis
Analysis examines retrieval efficiency, memory quality, and failure causes, showing that dynamic cutoff reduces context while evidence selection remains a key bottleneck.
- Efficiency Analysis: Observation generation dominates ingestion cost at 1.55 seconds per selected chunk, while structural memory update and removed-chunk handling take 49 ms and 0.6 ms per rotation.Streaming allows subsequent chunks to arrive while the current selected chunk is processed.
- Dynamic Retrieval Analysis: A short high-similarity prefix is followed by flatter scores, allowing dynamic cutoff to filter weakly related entries before hub-and-spoke expansion.This mechanism reduces marginal historical context and keeps retrieval compact.
- Dynamic Retrieval Analysis: Dynamic retrieval lowers the average number of retrieved tokens by about 51% on OVO-Bench.The reduction follows from cutting off retrieval before retaining all top-12 candidates.
- Memory Quality Analysis: Human evaluation assesses entity merging across 100 mention pairs, 170 entity hubs, and 96 spoke-to-hub attachments sampled from 30 StreamingBench videos.The mention pairs span 90 distinct hubs.
- Failure Cases Analysis: Among 100 incorrect predictions, answer synthesis accounts for 31% of failures, observation generation 23%, retrieval 19%, and entity linking 10%.The analysis identifies evidence selection as the main bottleneck and notes that most failures are predictive questions.
6 Conclusion
The conclusion presents D-HSM as a training-free balance between long-range memory and immediate perception, using structured, question-aware reconstruction of the past.
- Conclusion: D-HSM stores distant history as entity-centered hub-and-spoke textual memory while retaining recent frames for fine-grained perception.Its dynamic retrieval selects and expands question-relevant evidence on demand.
- Conclusion: Experiments on streaming and offline long-video benchmarks show that D-HSM outperforms existing online and offline baselines.The conclusion attributes the approach’s value to reconstructing the past through structured, question-aware memory.
7 Limitations
D-HSM’s textual memory depends on frozen-VLM-generated observations, creating risks of propagated recognition and description errors and loss of distant fine-grained detail.
- Limitations: Errors in entity recognition, OCR, or action descriptions may propagate into the structured memory.Because distant history is stored as text, fine-grained details from earlier moments may also be lost.
8 Ethical Considerations
The paper reports responsible-use considerations and describes the structured observations, memory construction, and streaming inference components used by D-HSM.
- Ethical considerations: The work uses public datasets and open-source models and does not collect private user data or involve human subjects.
- Ethical considerations: Potential risks include hallucinated or biased outputs and misuse of streaming video understanding systems in sensitive applications.
- Streaming inference: Streaming inference updates hub-and-spoke memory from selected chunks and combines retrieved textual evidence with a recent visual window for answer prediction.
- Structured observations: D-HSM converts selected historical chunks into fixed-schema textual observations rather than storing free-form captions.
- Structured observations: The observation schema records objects, people, actions, readable screen text, spatial relations, and a main event.
C More Ablation Experiments
The ablations show that D-HSM’s gains come from balancing recent visual perception with retrieved historical memory. The optimal balance depends on the benchmark’s temporal demands.
- Recent visual window size: Increasing recent frames consistently improves StreamingBench, but OVO-Bench peaks at four frames.This indicates that additional recent visual context is not always beneficial when backward or temporal evidence is required.
- Historical processing strategies: HSM Only outperforms Only Recent 4 Frames on OVO-Bench backward tasks, while the full method improves backward understanding and retains strong real-time performance.The comparison uses fixed HSM, recent frames, and Qwen2.5-VL+D-HSM (4f).
- Historical processing strategies: D-HSM integrates recent observations with long-term memory to improve temporal reasoning while preserving real-time performance.Recent-only processing is strong for real-time perception but limited on longer temporal reasoning and contextual consistency.
- Historical processing strategies: 82.45 overall score, up from 81.22, is achieved by Qwen2.5-VL+D-HSM (4f) on StreamingBench.The full method outperforms using only HSM or only the recent four frames, with gains on PR and CT.
- Retrieval context: D-HSM retrieves hub-and-spoke links and renders event, entity-centered action, and temporal action-chain evidence into textual context for the frozen VLM.The retrieval uses the recent visual window and question to construct the final context.
F Failure Analysis Breakdown
The failure analysis categorizes incorrect predictions by primary cause, separating observation, entity-linking, retrieval, and answer-synthesis errors.
- Failure categories: Failures are assigned one mutually exclusive primary cause: observation-generation, entity-linking, retrieval, or answer-synthesis errors.Observation errors involve visual recognition or OCR; the other categories concern memory structure, evidence selection, and frozen-VLM answering.
- Failure categories: Observation-generation errors cover incorrect visual recognition or OCR, while entity-linking errors merge distinct entities or split one entity across hubs.These categories identify errors in extracting observations and organizing entities in memory.
- Failure categories: Retrieval errors occur when required evidence is not selected or expanded, whereas answer-synthesis errors occur despite available context.The latter specifically concerns an incorrect answer from the frozen VLM.
- Failure analysis: Table F1 summarizes the primary causes of 100 manually analyzed failures.Indented rows decompose parent categories and are not counted as additional errors.
H Dataset Statistics
The supplied material identifies the evaluated video benchmarks and presents the algorithmic components used for hub-and-spoke memory construction and retrieval.
- Datasets: Evaluation includes StreamingBench, OVO-Bench, LongVideoBench, MLVU, and VideoMME.Only test splits are used because the method is training-free.
- Memory construction: Algorithm 2 updates hub-and-spoke memory using an entity-linking threshold and produces updated memory M_t.The supplied algorithm excerpt identifies the memory-update procedure and its threshold parameter.
- Memory construction: The memory representation combines hubs, spokes, co-occurrence edges, and temporal edges, with I(c) selecting entries supporting entity c.These structures provide the links used for organized historical evidence.
- Retrieval: Algorithm 3 retrieves a linearized textual context from memory using the question, retrieval budget K, threshold θ, and gap parameters.The retrieval procedure is explicitly defined to output C_t.
- Retrieval: Retrieval scores memory entries with cosine similarity between the encoded question and each entry, then selects top-K entries above threshold.The procedure sorts or selects entries according to similarity-based scores.