PaperlayerView exact revision on arXiv

Source-linked AI summary

Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams

Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, Xiaojie Jin

arXiv:2406.08085v2cs.CV

TL;DR

Existing video-language models are mainly designed for offline, short-video QA and struggle with real-time understanding of extremely long online streams. Flash-VStream addresses this with a STAR memory mechanism and asynchronous processing, achieving strong online and offline benchmark performance while reducing inference latency and VRAM consumption.

  • Problem

    Existing benchmarks and video-language models mainly target offline or short-video understanding, leaving real-time QA over extremely long online streams insufficiently addressed.

  • Method

    Flash-VStream uses STAR memory to compress and update visual information while asynchronously processing streaming frames and user questions through shared memory.

  • Results

    Flash-VStream achieves state-of-the-art performance on online benchmarks, maintains state-of-the-art performance on offline benchmarks, and significantly reduces inference latency and VRAM consumption.

  • Takeaways & Limitations

    VStream-QA provides a timestamped, substantially longer benchmark for evaluating online video stream understanding, while Flash-VStream demonstrates real-time long-stream QA capability.

Abstract

from arXiv · show

Benefiting from the advancements in large language models and cross-modal alignment, existing multi-modal video understanding methods have achieved prominent performance in offline scenario. However, online video streams, as one of the most common media forms in the real world, have seldom received attention. Compared to offline videos, the 'dynamic' nature of online video streams poses challenges for the direct application of existing models and introduces new problems, such as the storage of extremely long-term information, interaction between continuous visual content and 'asynchronous' user questions. Therefore, in this paper we present Flash-VStream, a video-language model that simulates the memory mechanism of human. Our model is able to process extremely long video streams in real-time and respond to user queries simultaneously. Compared to existing models, Flash-VStream achieves significant reductions in inference latency and VRAM consumption, which is intimately related to performing understanding of online streaming video. In addition, given that existing video understanding benchmarks predominantly concentrate on offline scenario, we propose VStream-QA, a novel question answering benchmark specifically designed for online video streaming understanding. Comparisons with popular existing methods on the proposed benchmark demonstrate the superiority of our method for such challenging setting. To verify the generalizability of our approach, we further evaluate it on existing video understanding benchmarks and achieves state-of-the-art performance in offline scenarios as well. All code, models, and datasets are available at the https://invinciblewyq.github.io/vstream-page/

1 Introduction

Flash-VStream addresses the difficulty of answering asynchronous questions over extremely long online video streams by combining continuous processing with compressed memory. The paper also introduces VStream-QA, an online-stream benchmark, and reports strong efficiency and benchmark performance.

  • Motivation: Existing video-language models struggle with real-time long-video question answering because redundant visual tokens exhaust VRAM and increase language-model decoding latency.Offline methods either retain all frame tokens, limiting input capacity, or encode only question-relevant content, which is unsuitable for asynchronous queries.
  • Flash-VStream: Flash-VStream mirrors a human-inspired “4-step, 2-process” pipeline, continuously perceiving and memorizing new frames while separately recalling information to answer user questions.Its frame encoder resembles the eyes, its LLM resembles the brain, and the two processes communicate through shared memory.
  • VStream-QA: VStream-QA marks each question-answer pair with a timestamp and restricts it to preceding visual information, matching online question answering.Its 30–60-minute videos are substantially longer than existing benchmarks and include ego-centric and third-person content.
  • Results: Flash-VStream achieves state-of-the-art performance on online benchmarks while reducing inference latency and VRAM consumption, and retains state-of-the-art performance on offline benchmarks.Figure 2 measures latency against frame number on an A100 GPU; the model answers user queries within 1 second.
  • Flash-VStream: STAR memory compresses necessary visual information, updates memory online, and omits redundancy between consecutive frames.The mechanism is designed for real-time memory writing and reading during streaming operation.

2 Related work

Related work spans multimodal large language models, real-time video stream tasks, and memory mechanisms for long-sequence processing. Flash-VStream focuses these directions on real-time natural-language question answering over video streams.

  • Multimodal large language models: Multimodal large language models combine text with visual data, and video-language research extends image-text modeling to video inputs.The cited model families include BLIP, LLaVA, and other video-oriented multimodal systems.
  • Real-time video stream understanding: Flash-VStream separates visual encoding from answer decoding in an online, real-time pipeline.This framing connects streaming video processing with asynchronous user interaction.
  • Real-time video stream understanding: Most existing real-time video methods target specific tasks such as object tracking or action recognition, whereas this work addresses natural-language video-stream question answering.The paper characterizes question answering as a more comprehensive task because natural language provides a general interface across tasks and modalities.
  • Memory mechanism for long sequence processing: Memory mechanisms are used across sequence-processing tasks, and prior work includes learnable visual memory inspired by the working-memory system of human cognition.The related applications include time-series forecasting, recommendation, machine translation, and video object segmentation.

3 Flash-VStream

Flash-VStream combines a streaming visual encoder, STAR memory, and LLM decoder in two asynchronous processes, enabling continuous frame processing and real-time question answering. STAR compresses information at spatial, temporal, abstract, and retrieved granularities while limiting memory size.

  • Framework overview: Flash-VStream uses a streaming visual encoder, STAR memory with a feature buffer, and an LLM decoder deployed across asynchronous frame-handler and question-handler processes.The frame handler encodes frames and consolidates memory, while the question handler reads shared memory and answers user questions.
  • Streaming visual encoder: The streaming visual encoder continuously maps each video frame to a feature map of ViT patch tokens.The encoder maps V_t ∈ R^(H×W×3) to e_t ∈ R^(P×P×D).
  • STAR memory: STAR memory stores information at four granularities: recent spatial details, clustered temporal contexts, abstract semantics, and retrieved key-frame features.Spatial memory uses FIFO updates; temporal memory uses Weighted K-means; abstract memory uses Semantic Attention; retrieved memory supplements temporal context with nearest frame features.
  • STAR memory: Spatial memory retains recent detailed features, while temporal memory condenses accumulated content into cluster centroids representing key video events.Temporal consolidation is triggered when memory exceeds N_tem and produces N_tem clusters.
  • STAR memory: Abstract memory synthesizes spatial and temporal information into a synopsis of the whole video, and retrieved memory recalls precise spatial details from selected clusters.Retrieved memory selects the top-K largest temporal clusters and retrieves nearby frame features from the feature buffer.
  • Real-time question answering: The question handler embeds each query, projects STAR memory into the embedding space, and generates an answer with the LLM decoder in real time.The implementation uses CLIP ViT-L/14-224px, a 2-layer MLP projector, and Vicuna-7B; STAR memory is limited to 681 tokens.

4 VStream-QA: A new benchmark for online video stream QA

VStream-QA is designed to evaluate online video-stream question answering, addressing the short, offline focus of earlier benchmarks. It combines timestamp-grounded questions with long, diverse videos and balanced question types.

  • Motivation: Existing video-QA benchmarks mainly evaluate offline understanding and typically use videos shorter than 10 minutes, leaving online long-stream evaluation underrepresented.Offline benchmarks provide the query and finite video together, unlike online stream understanding.
  • Benchmark design: VStream-QA pairs each question with a video timestamp and restricts its answer to visual information available before that timestamp.This setting is intended to match online video-stream understanding.
  • Question types: The benchmark covers diverse video sources and five balanced question types targeting scene content, actions, event occurrence, and temporal ordering.The types include Scene Summary, Action Description, Event Occurrence, Ordered Event Narrative, and Sequence Validation.
  • Benchmark composition: VStream-QA contains 10 one-hour ego-centric clips and 22 half-hour movie clips, totaling 21 hours of video and 3.5K question-answer pairs.VStream-QA-Ego contains 1.5K triplets, while VStream-QA-Movie contains 2K triplets.
  • Data generation: The dataset is generated through video selection, dense captioning, summary generation, question-answer generation, and human filtering.The five-step pipeline balances annotation quality, dataset scale, and annotation expenses.

5 Experiment

Experiments evaluate Flash-VStream on real-time and offline video QA, computational efficiency, memory design, and qualitative long-video understanding. The model achieves strong benchmark performance while reducing latency and VRAM costs.

  • Evaluation setup: Flash-VStream is evaluated on real-time and conventional zero-shot video question-answering datasets against several competitive video-language models.The comparisons include RVS-Ego/Movie and normal zero-shot VideoQA benchmarks.
  • Main results: Flash-VStream achieves state-of-the-art performance on online benchmarks while reducing inference latency and VRAM consumption.The reported efficiency advantage is attributed to multiprocessing STAR memory for streaming frame processing rather than processing all frames at once.
  • Memory ablations: Removing temporal memory causes a severe performance drop, indicating that it integrates contextual information across frames for coherent long-video comprehension.Spatial, abstract, and retrieved memory also contribute by capturing different aspects of visual information.
  • Memory ablations: Semantic Attention outperforms other memory-updating strategies by a large margin and selectively forgets irrelevant details while extracting key information.The authors associate this advantage with extracting key information and supporting abstract concepts in long videos.
  • Additional analyses: Balanced spatial and temporal memory sizes are preferred because both undersized and excessively large or long memories can harm performance or increase computational cost.Figure 6 visualizes memory-token consolidation across ActivityNet and Ego4D, while the case study shows Flash-VStream uniquely recognizing a long movie clip’s theme.

6 Conclusion

The paper presents Flash-VStream for real-time online video processing and VStream-QA for online video understanding evaluation. It reports lower latency and VRAM use, superior online-benchmark performance, and state-of-the-art offline performance.

  • Contributions: Flash-VStream processes online video streams in real time and answers user questions using the STAR memory mechanism.The model is designed for real-time processing and question answering over online streams.
  • Contributions: VStream-QA is introduced as a benchmark for online video understanding.The benchmark complements the model contribution by evaluating online-stream understanding.
  • Results: Flash-VStream significantly reduces inference latency and VRAM consumption.The conclusion reports these reductions without specifying numerical values in the supplied passage.
  • Results: The model outperforms existing methods on the new online benchmark and maintains state-of-the-art performance on offline video-understanding benchmarks.These are the paper’s reported online and offline evaluation outcomes.

A Memory implementation details

The STAR memory implementation combines spatial, temporal, abstract, and retrieved memory to compress, update, and recall visual information at different granularities.

  • Memory components: STAR memory uses parametric and non-parametric updating strategies, with spatial memory employing a simple replacing method.The mechanism is organized around multiple memory types and a feature buffer.
  • Temporal memory: Temporal memory applies weighted K-means clustering when its capacity is exceeded, condensing content into centroids representing key video events.Each temporal-memory feature represents a cluster centroid.
  • Temporal memory: Weighted K-means assigns features to the nearest centroids, updates weighted centroids iteratively, and returns the condensed temporal memory and cluster weights.The procedure stops when assignments stabilize or the iteration limit is reached.
  • Abstract memory: Abstract memory uses Semantic Attention to integrate spatial and temporal information into a synopsis that is updated using newest features.The update uses attention weights, softmax normalization, and momentum with decay factor α.
  • Retrieved memory: Retrieved memory selects the largest temporal clusters and retrieves nearby frame features from the buffer to restore precise spatial details.The number of retrieved clusters is K = Nret, and selected features supplement temporal memory.

B Training details

Flash-VStream is trained in two stages that update the Semantic Attention model, projector, and LLM. With extracted visual features, training takes 15 hours on 8 A100 80G GPUs.

  • Training procedure: The training procedure has a modality-alignment stage followed by an instruction-tuning stage.The first stage trains the Semantic Attention model and projector; the second fine-tunes those components and the LLM.
  • Training procedure: 15 hours on 8 A100 80G GPUs is sufficient to complete training with extracted visual features.Training uses BFloat16, and detailed settings are reported in Table 7.

C VStream-QA benchmark design details

This section introduces the VStream-QA benchmark as an online video understanding resource.

  • VStream-QA is presented as a benchmark for online video understanding.

C.1 Data generation pipeline in detail

VStream-QA is constructed from diverse long videos through captioning, summarization, timestamped question generation, and human filtering. The benchmark is intended to evaluate online streaming understanding, while remaining narrower than fully general real-world comprehension.

  • Video Selection: The videos include ego-centric and movie content selected to cover diverse video types.The selection uses 10 one-hour Ego4D videos and 22 thirty-minute MovieNet videos.
  • Dense Captioning: Dense captioning divides long videos into 30-second pieces, sparsely samples eight frames per piece, and assigns timestamps to generated captions.
  • Summary Generation: GPT-4 deduplicates dense captions into concise scene-level summaries while preserving timestamps.
  • Question-Answer Generation: GPT-4 generates five question-answer types from one or more consecutive scene summaries, ensuring questions use only information before their timestamps.
  • Human Filtering: Human filtering removes irrelevant, ambiguous, externally dependent, video-independent, incorrect, and repetitive question-answer pairs.
  • Scope: VStream-QA does not fully represent infinitely long streams and covers only coarse-grained question answering rather than more complex comprehension tasks.

D.2 GPT-3.5-based evaluation metric

The section reports a credibility problem in GPT-3.5-based evaluation: score distributions can disagree with accuracy, particularly for answers classified as “no.” It also notes broader societal risks associated with real-time long-video understanding models.

  • Broader impacts: Real-time long-video understanding models may enable unauthorized surveillance or privacy-infringing tracking.
Loading 2406.08085v2…