PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamEMS: Streaming Video Understanding with Self-Evolving Memory Scheme for Vision-Language Models

Yuxin Liu, Peiqin Zhuang, Yali Wang

arXiv:2608.27881v1cs.CV

TL;DR

Streaming video methods often optimize memory read/write operations while leaving the representational capability of stored memory underused. StreamEMS self-evolves historical memory through semantic aggregation and prior-informed refinement, and it outperforms other methods on OVO-Bench and StreamingBench, including high token-drop settings.

  • Problem

    Existing streaming video methods focus on external memory read/write operations and overlook opportunities to enhance memory's own representational capability.

  • Method

    StreamEMS restructures historical memory with a Semantic Evolution Module for information-dense representations and a Prior-informed Evolution Module for EMA-based refinement using prior memory distributions.

  • Results

    StreamEMS outperforms other methods on OVO-Bench and StreamingBench, with its advantage remaining evident under high token usage drop rates.

  • Takeaways & Limitations

    Self-evolving memory provides a complementary way to exploit memory internally for streaming video understanding.

  • Takeaways & Limitations

    The higher semantic threshold criterion may discard potentially useful memory entities and compromise the overall memory representation.

Abstract

from arXiv · show

Recently, many streaming video understanding methods have been proposed by constructing an external memory to store historical data for computational reduction. Most methods focus on optimizing the injection procedure of current data (write) and retrieving informative historical data (read) from memory, while overlooking the opportunity to further enhancing the representational capability of memory itself. In this work, we present StreamEMS, a general mechanism for improving streaming video understanding by re-structuring the historical data stored in memory through self-evolving memory scheme, enabling more informative and robust memory representations. Specifically, we first introduce a Semantic Evolution Module to evolve the memory into more information-dense representations by exploiting informative memory entities discovered via progressively shrinking semantic scales from coarse to fine. In addition, we further introduce a Prior-informed Evolution Module to evolve memory into more robust representations by leveraging prior memory distributions to refine the current memory state. We validate the effectiveness of our proposed designs on widely-used streaming video understanding datasets, i.e., OVO-Bench and StreamingBench, and the results showcase that our method performs better than other methods. Moreover, the advantage of our method becomes consistently evident even under high token usage drop rate settings, indicating the effectiveness and robustness of our method in unleashing the potential of the memory itself.

1. Introduction

Streaming video understanding methods commonly optimize external memory read/write operations, but StreamEMS instead restructures stored historical data to improve memory representations. It introduces semantic and prior-informed evolution modules and reports stronger performance on OVO-Bench and StreamingBench, including high token-drop settings.

  • Research gap: Existing methods mainly optimize external memory read/write operations, overlooking the intrinsic representational capacity of memory itself.Write injects current data, while read retrieves historical data.
  • Proposed framework: StreamEMS restructures historical memory data through self-evolution to mine distinctive cues for subsequent reasoning.The approach is presented as complementary to prior memory operations.
  • Evaluation: StreamEMS outperforms other methods on OVO-Bench and StreamingBench, with its advantage remaining evident under high token usage drop rates.The authors interpret this as evidence of effectiveness and robustness in exploiting memory itself.
  • Semantic Evolution Module: The Semantic Evolution Module uses semantic graphs and progressively shrinking coarse-to-fine scales to form more information-dense memory representations.It exploits informative semantic entities and propagates information within memory.
  • Prior-informed Evolution Module: The Prior-informed Evolution Module uses EMA to integrate prior memory distributions and refine the current memory state into more robust representations.This balances current and historical importance distributions and reduces reliance on the current distribution alone.

2. Related Work

Related streaming video methods primarily construct or operate on memory through read/write mechanisms, while self-evolving memory had not been examined for streaming video understanding. StreamEMS adapts internal information refinement and temporal-prior strategies to boost native memory.

  • Memory-based streaming methods: Streaming video methods use external or implicit memory to compress lengthy visual context and support later understanding.Common strategies include token dropping, token merging, and improved memory reading.
  • Internal memory evolution: Prior approaches mainly center on external memory interactions, whereas StreamEMS explores the intrinsic capacity of memory itself.The authors describe the proposed method as complementary to existing external-memory techniques.
  • Self-evolving schemes: Self-evolving schemes using prior predictions and EMA have been studied elsewhere, but had not been examined in streaming video understanding.StreamEMS formulates these paradigms for the streaming video setting.

3. Method

StreamEMS extends the standard vision-language streaming pipeline by evolving memory entities internally. SEM progressively propagates information among relevant entities, while PEM integrates historical similarity distributions to refine current memory representations.

  • Overview: The standard pipeline uses vision and text encoders to produce multimodal inputs, then an LLM reasons over them with compressed visual context in external memory.StreamEMS further restructures the memory entities inside that pipeline.
  • Semantic Evolution Module: SEM constructs affinity-based semantic graphs and uses masked self-attention to propagate information from relevant key/value entities to query entities.The affinity matrix is based on cosine similarity between memory entities.
  • Memory construction: Initial memory entities are clip-level features built from buffered frame-level features and learnable queries, summarizing long-range context with lower computational overhead than frame-by-frame processing.The frame buffer selectively retains visually similar frame features before aggregation.
  • Semantic Evolution Module: SEM progressively tightens the similarity threshold so nodes aggregate increasingly relevant neighbors, producing more compact and information-dense memory.The strategy moves through progressively shrinking semantic scales from coarse to fine.
  • Prior-informed Evolution Module: PEM integrates distinct information from the previous memory state by applying EMA to historical similarity distributions and refining the current memory state.A smoothing coefficient balances prior and current similarity distributions.

4. Experiments

Experiments show that StreamEMS improves streaming and offline video understanding while reducing memory and inference costs. Its self-evolving modules remain effective under aggressive token reduction and complement each other.

  • Online Video Understanding: StreamEMS improves OVO-Bench accuracy from 47.6% to 51.4% over TimeChat-Online.It also reports average accuracies of 65.8% for Real-Time Visual Perception and 48.21% for Backward Tracing.
  • Online Video Understanding: 51.41% vs. 49.4% accuracy separates StreamEMS from StreamAgent.The comparison is presented as evidence that internal memory refinement can outperform the agentic StreamAgent model.
  • Online Video Understanding: StreamEMS outperforms other methods on most StreamingBench subtasks, demonstrating generalization across datasets.The reported evaluation covers StreamingBench performance and different streaming video understanding tasks.
  • Offline Video Understanding: StreamEMS achieves 65.2% on MVBench, surpassing TimeChat-Online by 4.8%.It also reports 76.8% on TempCompass Multi-choice, up from 70.9%, and 51.3% on VideoMME.
  • Ablation Study: The combined SEM and PEM modules yield a 2.95% improvement over the corresponding baseline configuration.The results indicate complementary benefits from semantic information propagation and temporal priors from previous memory states.
  • Token Efficiency: StreamEMS consistently outperforms TimeChat-Online as token dropping increases, with a 5.81% margin at 70%–85% dropping rates on OVO-Bench.It maintains a 2.12% margin at 85%–95% dropping rates, indicating higher token efficiency under limited-token settings.

5. Visualization

The visualization illustrates how StreamEMS progressively refines memory connections and uses prior distributions to improve temporal relevance. These changes yield more discriminative representations and a correct prediction in the illustrated example.

  • Semantic Evolution: Progressively stricter semantic thresholds concentrate the anchor clip’s connections on semantically relevant clips.Distinct information from neighboring clips is iteratively distilled into the anchor clip, producing a more compact and informative memory representation.
  • Prior-informed Evolution: The original similarity distribution can underweight temporally adjacent clips while favoring temporally distant clips.The visualization associates this temporal bias with attention to irrelevant clips.
  • Prior-informed Evolution: Prior distributions recalibrate the current similarity distribution toward the true temporal context.The example contrasts the baseline’s irrelevant focus, such as adding sugar, with StreamEMS’s correct answer after memory evolution.
  • Qualitative Result: The self-evolving memory schemes produce a more discriminative representation that leads to the correct prediction in the illustrated question.The qualitative result is presented as validation of the proposed approach.

6. Conclusion

StreamEMS improves streaming video understanding by restructuring historical memory through a self-evolving scheme that produces more informative and robust representations. Its Semantic Evolution Module and Prior-informed Evolution Module support these improvements, and experiments show effectiveness on streaming video understanding datasets.

  • StreamEMS restructures historical data in memory through a self-evolving scheme for streaming video understanding.
  • The Semantic Evolution Module evolves memory into more information-dense representations by exploiting informative memory entities.It uses hierarchical semantic graphs built with coarse-to-fine semantic thresholds.
  • The Prior-informed Evolution Module evolves memory into more robust representations by using prior memory distributions to refine the current memory state.
  • StreamEMS was validated on widely used streaming video understanding datasets, and the results demonstrate the effectiveness of its designs.

7. Declaration of Generative AI and AI-Assisted Technologies in the Manuscript Preparation Process

The authors used large language models during manuscript preparation for grammar correction, language refinement, and stylistic improvements. They reviewed and edited the output and retained responsibility for the published content.

  • Large Language Models were used solely for grammar correction, language refinement, and stylistic improvements.
  • The authors reviewed and edited the generated output as needed during manuscript preparation.
  • The authors take full responsibility for the content of the published article.
Loading 2608.27881v1…