PaperlayerView exact revision on arXiv

Source-linked AI summary

Flash-VStream: Efficient Real-Time Understanding for Long Video Streams

Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Xiaojie Jin

arXiv:2506.23825v2cs.CV

TL;DR

Long-video understanding is hindered by substantial computational and memory overhead, making real-time question answering difficult. Flash-VStream uses an asynchronous two-process architecture with Flash Memory that aggregates temporal context and retrieves detailed key-frame information. It reduces inference latency while delivering strong performance across long-video and comprehensive video benchmarks.

  • Problem

    Long-video understanding remains difficult because long-context processing imposes substantial computational and memory overhead, while existing approaches are inefficient for real-world applications.

  • Method

    Flash-VStream separates continuous frame processing from question answering and uses Context Synopsis Memory plus Detail Augmentation Memory to represent long-term context and detailed key frames.

  • Results

    Flash-VStream achieves superior understanding across challenging video benchmarks, saves 53% of video tokens versus Qwen2-VL on MLVU, and reduces latency to meet real-time standards.

  • Takeaways & Limitations

    The fixed-size Flash Memory and asynchronous framework enable efficient processing of extremely long video streams with real-time responses.

Abstract

from arXiv · show

Benefiting from the advances in large language models and cross-modal alignment, existing multimodal large language models have achieved prominent performance in image and short video understanding. However, the understanding of long videos is still challenging, as their long-context nature results in significant computational and memory overhead. Most existing work treats long videos in the same way as short videos, which is inefficient for real-world applications and hard to generalize to even longer videos. To address these issues, we propose Flash-VStream, an efficient video language model capable of processing extremely long videos and responding to user queries in real time. Particularly, we design a Flash Memory module, containing a low-capacity context memory to aggregate long-context temporal information and model the distribution of information density, and a high-capacity augmentation memory to retrieve detailed spatial information based on this distribution. Compared to existing models, Flash-VStream achieves significant reductions in inference latency. Extensive experiments on long video benchmarks and comprehensive video benchmarks, i.e., EgoSchema, MLVU, LVBench, MVBench and Video-MME, demonstrate the state-of-the-art performance and outstanding efficiency of our method. Code is available at https://github.com/IVGSZ/Flash-VStream.

1. Introduction

Long video understanding remains challenging because its computational and memory demands hinder practical, real-time use. Flash-VStream addresses this with asynchronous processing and Flash Memory, achieving efficient real-time understanding.

  • Motivation: Long video understanding is computationally and memory intensive, limiting deployment on resource-constrained devices and increasing inference latency.The paper defines real-time VLMs as responding within one second.
  • Motivation: Existing state-of-the-art VLMs struggle to respond in real time when answering questions about long videos.This limitation affects multimodal assistants, robots, and surveillance systems requiring rapid interaction.
  • Approach: Flash Memory combines Context Synopsis Memory for temporal information density with Detail Augmentation Memory for detailed key-frame content.The design allocates more computation to informative frames while preserving long-term context.
  • Approach: Flash-VStream uses a two-process framework that continuously encodes frames and asynchronously generates answers from shared Flash Memory.The frame handler updates memory while the question handler responds to incoming queries and generates the first answer token within one second.
  • Results: Flash-VStream significantly reduces inference latency, balances accuracy and efficiency, and achieves state-of-the-art performance on the full EgoSchema benchmark.The paper reports this result in comparison with prior methods.

2. Related Work

Prior video-language models compress or select video content effectively for short videos, but long-video performance and real-time free-form question answering remain challenging. Flash-VStream combines streaming processes with memory-based temporal and spatial representation.

  • Video Language Models: Many video-language models reduce video representation with compressed or dynamic tokens, but these methods have relatively poor performance on long-video benchmarks.Examples include representing single-frame features with two tokens and compressing entire videos into fewer tokens.
  • Video Language Models: Question-aware methods select question-related clips or key frames, whereas Flash-VStream aggregates information from visual content without relying on specific questions.This supports general information aggregation before user queries arrive.
  • Real-Time Video Stream Understanding: Real-time video understanding has primarily targeted specific tasks such as object tracking, action recognition, segmentation, narration, and action anticipation.Flash-VStream instead focuses on free-form video question answering.
  • Real-Time Video Stream Understanding: Flash-VStream separates continuous vision processing from online language processing through a frame handler and a question handler sharing Flash Memory.The frame handler encodes and consolidates frames while the question handler waits for user queries.
  • Memory Mechanisms: Memory mechanisms have been applied to long-sequence and video-understanding tasks, including long-term, short-term, consolidated, and multi-granularity memories.The paper positions Flash Memory within this broader line of memory-based video modeling.

3. Flash-VStream

Flash-VStream uses Flash Memory to compress long-term temporal information while selectively preserving detailed spatial features from key frames. Its asynchronous two-process design and bounded token budget support efficient real-time video understanding.

  • Flash Memory: Flash Memory combines Context Synopsis Memory for temporal aggregation with Detail Augmentation Memory for fine-grained spatial details.CSM maintains compressed cluster centroids, while DAM stores high-resolution feature maps from selected key frames.
  • Model Architecture: The frame handler continuously encodes frames and updates shared Flash Memory, while the question handler answers queries concurrently.This decouples vision processing from language processing and allows the first answer token to be generated within one second.
  • Context Synopsis Memory: CSM represents semantically similar frames with low-resolution cluster centroids, whose cluster sizes implicitly encode information density.K-means updates the memory while limiting the number of clusters to N_CSM.
  • Detail Augmentation Memory: DAM retrieves high-resolution spatial features by using large CSM cluster centroids as anchors for key-frame localization.A frame is selected when its low-resolution feature is nearest to a centroid anchor in feature space.
  • Adaptive Multimodal RoPE: Flash Memory interleaves CSM and DAM tokens by temporal position, with AM-RoPE assigning flexible positional representations to the two token types.The memory contains N_CSM × h′ × w′ plus N_DAM × h × w video tokens, and limiting NVtokens to 12000 enables real-time inference for a 7b model.

4. Experiments

Flash-VStream is evaluated across long and comprehensive video benchmarks, with experiments targeting real-time latency, equal-cost comparisons, and the contributions of its memory components. Ablations further examine clustering, retrieval, selection, and memory consolidation choices.

  • Evaluation Settings: Zero-shot multiple-choice VQA covers three long-video and two comprehensive video benchmarks, including videos ranging from 11 seconds to 1 hour.The benchmarks are EgoSchema, MLVU, LVBench, MVBench, and Video-MME.
  • Computational Efficiency: Flash-VStream achieves real-time response with NVtokens <= 12000, while several competing models exceed one second at higher token costs.Latency is measured on a single A100 GPU using BFloat16 precision and FlashAttention-2.
  • Main Results: Flash-VStream generally outperforms competitive models across challenging benchmarks, while saving 53% of video tokens compared with Qwen2-VL on MLVU.It performs slightly behind Qwen2-VL on MLVU but is reported as more efficient.
  • Main Results: Under equal-cost testing, Flash-VStream significantly surpasses Qwen2-VL-online on both short-video and long-video benchmarks.Qwen2-VL-online is constrained to 11520 video tokens.
  • Flash Memory Ablation: CSM and DAM improve average accuracy by 2.0% and 0.7%, respectively, compared with uniform sampling.The ablation study evaluates the effects of removing each Flash Memory component.
  • Sampling and Retrieval Ablations: K-means surpasses uniform sampling by 3.4% on EgoSchema and 1.6% on MVBench, while Feature-Centric retrieval improves performance across short- and long-video benchmarks.Cosine Similarity performs relatively worse, and top-k largest selection is comparable to sampling all frames.

5. Conclusion

Flash-VStream is an efficient video-language model for long video streams that provides real-time responses. Its asynchronous processing and Flash Memory design support strong performance and efficiency across comprehensive video benchmarks.

  • Flash-VStream handles long video streams and provides real-time responses through an efficient video-language model.Its two-process asynchronous framework separates vision and language processing.
  • The Flash Memory module combines Context Synopsis Memory for temporal aggregation with Detail Augmentation Memory for detailed spatial retrieval.The design exploits temporal redundancy in videos.
  • Extensive experiments on multiple comprehensive video benchmarks show superior performance and efficiency compared with existing state-of-the-art models.

Supplementary Material

The supplementary material provides implementation details, efficiency analyses, additional memory-structure ablations, and more visual performance cases.

  • The supplement documents Flash Memory implementation and training settings.
  • It analyzes model inference efficiency and evaluates additional memory structure configurations through ablation studies.
  • Additional visual cases provide a more comprehensive view of model performance.

A. Implementation Details

Flash Memory consists of Context Synopsis Memory and Detail Augmentation Memory, using clustering and retrieval to compress temporal information and preserve informative spatial details.

  • Flash Memory comprises Context Synopsis Memory and Detail Augmentation Memory with clustering-based and retrieval-based updating policies.
  • Context Synopsis Memory aggregates long-context temporal information and models information-density distribution.
  • Temporal K-means clustering condenses (N_CSM + 1) × h′ × w′ tokens into N_CSM × h′ × w′ tokens.
  • Detail Augmentation Memory stores spatial details from the most informative key frames using feature-centric sampling.
  • Feature-centric retrieval selects centroids of the top-k largest clusters as anchors and adds nearest high-resolution features to Detail Augmentation Memory.Cluster weights equal the numbers of feature maps in the clusters.

B. Training Details

Flash-VStream is trained with a compact LoRA fine-tuning setup, while efficiency evaluations use A100 GPUs with BFloat16 and FlashAttention-2.

  • Flash-VStream is trained for one epoch on a 9k subset of LLaVA-Video.The visual encoder is frozen, while projector and LLM linear layers are LoRA fine-tuned.
  • Figure 6 compares response latency, maximum GPU memory, and execution wall time.The experiments use A100 GPUs, BFloat16, and FlashAttention-2.
  • Table 7 reports the training settings for Flash-VStream.
  • Training finishes in about 10 hours on 8 A100 80G GPUs using BFloat16 automatic mixed precision and FlashAttention-2.

C. Efficiency Analysis

Flash-VStream offers lower response latency and GPU memory consumption than competing video language models. Its question handler remains under one second as input length grows because it uses fixed-size Flash Memory.

  • Flash-VStream demonstrates superior response latency and maximum GPU memory efficiency compared with competitive video language models.Response latency remains below one second, while memory consumption stays relatively stable as input frames increase.
  • < 1s question-handler execution is maintained regardless of the number of input frames.The question handler relies only on size-fixed Flash Memory.
  • More than 1 second of frame-handler time appears when the number of frames exceeds 1000.This can delay visual-information updates but does not affect response latency.
  • Flash-VStream’s efficiency advantages in latency and GPU memory make it suitable for real-time long-video understanding tasks.

D. Ablation Study on Memory Structure

The ablation study examines memory-structure sensitivity and reports accuracy scaling with memory size. Separate case studies illustrate broad understanding capabilities, while fail cases identify difficult edited-video scenarios.

  • Ablation Study on Memory Structure: Accuracy scales with different choices of the CSM and DAM memory sizes.Table 8 compares cluster numbers for CSM and key-frame numbers for DAM against baseline configurations.
  • Ablation Study on Memory Structure: The grid-search configuration is supported by the observed memory-structure scaling trend.
  • Case Study: Flash-VStream demonstrates object, action, temporal, counting, and reasoning capabilities across documentary, cartoon, commercial, sports, and tutorial videos.
  • Fail Case Analysis: Flash-VStream may produce incorrect predictions on text-intensive long videos and videos with rapid scene changes.The paper attributes these difficulties to information-density distributions that differ from those of native videos.

F.2. GPT-3.5-based Metric for Open-ended VQA

The paper questions GPT-3.5-based judgment for open-ended VQA because its scores can be unstable, biased, and difficult to reproduce. Case studies instead present ground-truth answers alongside predictions from three video-language models.

  • GPT-3.5-based Metric: GPT-3.5-based accuracy and score metrics are described as unstable and prone to bias.Proprietary APIs, upgrades over time, and GPT hallucinations undermine reliability, stability, and reproducibility.
  • GPT-3.5-based Metric: A discrepancy between GPT accuracy and GPT score distributions remains across the evaluation.
  • Case Studies: The case studies compare Qwen2-VL, LLaVA-OV, and Flash-VStream predictions against indicated ground-truth answers across varied videos.Examples include documentaries, cartoons, advertising, sports, tutorials, animals, and space-debris content.
  • Fail Case Analysis: The fail-case figures include questions about space-debris cleanup and a white parrot’s action.
Loading 2506.23825v2…