Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

Hongyu Qu; Guangming Yao; Ling Xing; Xiaobin Hu; Rongxing Ding; Guibin Zhang; Fan Zhang; Yi Yuan; Xiangbo Shu; Shuicheng Yan

Source-linked AI summary

Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

Hongyu Qu, Guangming Yao, Ling Xing, Xiaobin Hu, Rongxing Ding, Guibin Zhang, Fan Zhang, Yi Yuan, Xiangbo Shu, Shuicheng Yan

arXiv:2609.04131v1Sep 3, 2026cs.CV

TL;DR

Streaming video models must answer queries over continuous visual input while operating causally and within bounded memory, yet retrieval-based methods leave history as external context. LatentStream addresses this gap by progressively internalizing retrieved evidence into evolving latent memory, and it reports state-of-the-art or highly competitive results across online and offline benchmarks.

Problem
Existing streaming methods retrieve historical evidence as external, variable-length visual context rather than internalizing it into a compact latent state for continued reasoning.
Method
LatentStream combines hierarchical streaming memory, progressively expanding latent memory evolution, and confidence-guided optimization to retrieve and internalize history under a fixed memory budget.
Results
64.2% on OVO-Bench, 76.9% on StreamingBench, 66.6% on VideoMME, 74.0% on MLVU, and 62.1% on LongVideoBench demonstrate state-of-the-art or highly competitive performance across online and offline tasks.
Takeaways & Limitations
Retrieve-and-internalize memory is presented as a promising direction for bridging external memory with latent reasoning in streaming Video-LLMs.

Abstract

from arXiv · show

Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize. Specifically, LatentStream comprises three coordinated components. First, Query-agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into a compact, fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.

1 Introduction

Streaming video understanding must handle continuously arriving visual inputs and real-time queries under bounded memory, but existing retrieval-centric methods leave accessed evidence as external context. LatentStream instead internalizes task-relevant history into an evolving latent memory through hierarchical storage, progressive retrieval, and confidence-guided optimization.

Motivation: Streaming video understanding requires causal processing of incoming observations and responses to queries posed at any time.The capability is relevant to live monitoring, autonomous driving, smart glasses, embodied agents, and robotic systems.
Existing approaches: Existing methods manage accumulated visual context through selection, token compression, cache management, or fixed-capacity and hierarchical memory, increasingly paired with query-aware retrieval.These training-free approaches adapt existing Video-LLMs without additional parameter updates.
Research gap: Retrieval-centric methods identify relevant history but expose it as variable-length external visual context rather than internalizing it into a compact state for subsequent reasoning.This leaves the relationship between query-agnostic memory and query-conditioned reasoning loosely connected.
LatentStream: LatentStream progressively internalizes task-relevant historical evidence into a compact, evolving latent memory that continuously guides streaming reasoning.The latent state evolves with newly accessed evidence and helps guide further retrieval, forming a retrieve-and-internalize paradigm.
Core components: LatentStream combines Query-agnostic Hierarchical Streaming Memory, Hierarchical Latent Memory Evolution, and Progressive Confidence-guided Latent Memory Optimization.HSM organizes observations into short-, mid-, and long-term memories under a fixed budget; HME uses expanding receptive fields; PMO uses group-wise predictive entropy.
Results: 64.2% on OVO-Bench, 76.9% on StreamingBench, 66.6% on VideoMME, 74.0% on MLVU, and 62.1% on LongVideoBench were achieved across online and offline evaluation.The reported results are under a bounded memory budget.

2 Related Work

Related work addresses streaming video understanding through causal interaction and long-term memory management, while LatentStream adds latent internalization of retrieved evidence. Its latent memory tokens evolve iteratively and guide subsequent historical access.

Streaming Video Understanding: Streaming video understanding differs from offline understanding by requiring causal processing of continuously arriving observations and real-time responses.Existing studies are categorized into four groups, including proactive interaction methods that determine when to respond.
Long-term Memory Management in Streaming Videos: Long-term memory management preserves useful historical information from unbounded visual streams under bounded memory and context budgets.Hierarchical multi-level memory organizes observations across temporal scales or granularities, retaining recent detail while consolidating older information.
LatentStream: LatentStream internalizes retrieved evidence into compact latent memory tokens that iteratively evolve and guide subsequent memory access.This distinguishes the method from approaches that keep historical evidence as external context.

3 Methodology

LatentStream builds a bounded hierarchical memory online, then progressively retrieves and internalizes query-relevant evidence into evolving latent memory tokens. Its groups access increasingly broad temporal scopes, while confidence-based optimization jointly refines latent tokens and retrieved evidence.

Query-agnostic Hierarchical Streaming Memory: Query-agnostic Hierarchical Streaming Memory organizes incoming visual tokens into short-, mid-, and long-term levels under a fixed token budget.Recent observations are densely retained, while older evidence is progressively consolidated across levels.
Query-agnostic Hierarchical Streaming Memory: Jenks-guided consolidation partitions temporal importance into drop, compress, and preserve groups before spatially merging redundant long-term tokens.Temporal scores determine short-to-mid transitions; spatial distances determine mid-to-long merging or individual retention.
Hierarchical Latent Memory Evolution: Three latent memory token groups use nested receptive fields spanning short-term, short-plus-mid-term, and complete memory-bank histories.A bootstrap pass contextualizes the initial tokens with the query and fixed external memory before iterative retrieval.
Hierarchical Latent Memory Evolution: At each evolution iteration, group-specific latent tokens refresh evidence retrieval from their assigned scopes, and the evolved tokens progressively adapt subsequent historical access.The initial latent state is query-conditioned through the bootstrap pass, while later relevance scores are recomputed after each update.
Progressive Confidence-guided Latent Memory Optimization: Retrieved evidence and latent memory tokens are jointly optimized through a confidence progression reward, internalizing task-relevant history without external supervision or parameter updates.Evidence is admitted only when the candidate reward improves, and group-wise entropy encourages increasing confidence as accessible history expands.

4 Experiment

Experiments evaluate LatentStream across online and offline video benchmarks, component ablations, hyperparameters, and efficiency. Results show consistent gains from hierarchical memory, latent internalization, progressive confidence optimization, and bounded-context processing.

Results on Streaming Video Benchmarks: LatentStream improves Qwen2.5-VL across streaming benchmarks while keeping the backbone frozen.On OVO-Bench, the 7B model reaches 64.2% overall, while StreamingBench reaches 76.9%.
Results on Offline Video Benchmarks: LatentStream achieves 66.6% on VideoMME, 74.0% on MLVU, and 62.1% on LongVideoBench, surpassing the Qwen2.5-VL-7B baseline by 3.3, 6.1, and 1.4 points.These results outperform compared training-free and training-based methods.
Key Component Analysis: The full model reaches 64.2% on OVO-Bench and 66.6% on VideoMME after HSM, PMO, and HME are added progressively.HSM improves the baseline, PMO adds further gains, and HME provides the best results by retrieving evidence from expanding memory scopes.
Effectiveness of Retrieve-and-internalize Latent Memory: Evolved latent memory reaches 64.2%/66.6% on OVO-Bench/VideoMME, outperforming direct evidence injection by 4.5%/1.0% points.Retrieved visual tokens are removed before final decoding, supporting retrieve-and-internalize rather than simple context augmentation.
Effect of Progressive Confidence-guided Latent Memory Optimization: The hierarchical progression reward reaches 64.2%/66.6%, outperforming absolute confidence optimization by 1.7%/1.1%.It additionally encourages the entropy ordering E_s > E_m > E_l.
Hyperparameter Analysis: Increasing evolution iterations from 0 to 4 raises average accuracy from 58.1% to 64.2%, after which further iterations provide no additional improvement.The study fixes R = 4 to balance performance and efficiency.
Hyperparameter Analysis: Increasing candidate budget B from 4 to 8 raises average accuracy from 63.0% to 64.2%, whereas larger budgets degrade performance.The results suggest that excessive candidates introduce redundancy that hinders optimization.
Hyperparameter Analysis: Increasing Gaussian noise scale σ from 1% to 10% improves exploration, while excessively large perturbations destabilize optimization and degrade performance.The study sets σ = 10%.

5 Conclusion

LatentStream replaces conventional store-and-retrieve memory with a compact, query-conditioned latent memory that progressively internalizes historical evidence. Across streaming and offline long-video benchmarks, the frozen-backbone framework demonstrates retrieve-and-internalize memory as a promising direction for streaming Video-LLMs.

Conclusion: LatentStream progressively internalizes task-relevant historical evidence into a compact, query-conditioned latent memory that guides streaming reasoning.It integrates hierarchical streaming memory, latent memory evolution, and confidence-guided optimization while keeping the underlying MLLM frozen.
Conclusion: Experiments across streaming and offline long-video benchmarks demonstrate LatentStream's effectiveness.The conclusion identifies retrieve-and-internalize memory as a promising direction for bridging external memory with latent reasoning.