PaperlayerView exact revision on arXiv

Source-linked AI summary

Streaming Video Question-Answering with In-context Video KV-Cache Retrieval

Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, Hao Jiang

arXiv:2503.00540v1cs.CV

TL;DR

Long-video VideoQA systems are inefficient for streaming because they process extensive frame context and may repeat processing for different queries. ReKV provides a training-free streaming framework that uses sliding-window encoding, stored KV-Caches, and query-relevant retrieval; experiments report improved accuracy with stable latency and memory usage, while limitations remain for extremely long streams and fixed retrieval choices.

  • Problem

    Long-video VideoQA is limited by large video-token volumes, sparse sampling, and repeated frame processing, which can lose fine-grained information and hinder streaming responses.

  • Method

    ReKV integrates with existing Video-LLMs without training by combining sliding-window attention, stored KV-Cache offloading, and external or internal query-relevant cache retrieval.

  • Results

    ReKV improves accuracy on long-form VideoQA and StreamingVQA benchmarks while reducing inference latency and memory usage and maintaining stable latency and GPU memory as frames increase.

  • Takeaways & Limitations

    ReKV supports efficient streaming VideoQA by preserving video context in KV-Caches and retrieving only information relevant to each question.

  • Takeaways & Limitations

    Extremely long streams may make RAM or disk KV-Cache size unsustainable, while fixed block sizes and fixed frame counts can disrupt continuity or limit retrieval flexibility.

Abstract

from arXiv · show

We propose ReKV, a novel training-free approach that enables efficient streaming video question-answering (StreamingVQA), by seamlessly integrating with existing Video Large Language Models (Video-LLMs). Traditional VideoQA systems struggle with long videos, as they must process entire videos before responding to queries, and repeat this process for each new question. In contrast, our approach analyzes long videos in a streaming manner, allowing for prompt responses as soon as user queries are received. Building on a common Video-LLM, we first incorporate a sliding-window attention mechanism, ensuring that input frames attend to a limited number of preceding frames, thereby reducing computational overhead. To prevent information loss, we store processed video key-value caches (KV-Caches) in RAM and disk, reloading them into GPU memory as needed. Additionally, we introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only query-relevant KV-Caches, ensuring both efficiency and accuracy in question answering. ReKV enables the separation of video encoding and question-answering across different processes and GPUs, significantly enhancing the efficiency of StreamingVQA. Through comprehensive experimentation, we validate the efficacy and practicality of our approach, which significantly boosts efficiency and enhances applicability over existing VideoQA models.

1 INTRODUCTION

StreamingVQA extends VideoQA to continuously processed long streams, requiring timely answers without repeatedly processing all past frames. ReKV addresses this with streaming encoding, KV-Cache retrieval, and decoupled processes while improving efficiency and accuracy.

  • Motivation: StreamingVQA requires continuous long-video processing and prompt responses to questions about previously viewed content.The task generalizes offline VideoQA, which processes the entire video and questions together.
  • Motivation: Current Video-LLMs struggle with long streams because large video-token volumes force sparse frame sampling or limit video length.These constraints can discard fine-grained visual information needed for complex question answering.
  • Approach: ReKV uses sliding-window attention for short-term context and retrieves query-relevant KV-Caches for long-term information without additional training.It supports external retrieval through CLIP-like models and internal retrieval through Video-LLM parameters.
  • Contributions: ReKV separates video encoding from question-answering and maintains stable inference latency and memory usage as frames increase.The paper reports improved accuracy on long-form and StreamingVQA benchmarks alongside reduced inference latency and memory usage.

2 STREAMINGVQA: TASK DEFINITION AND DISCUSSION

StreamingVQA asks models to answer questions at arbitrary times using only video observed so far, unlike offline VideoQA over pre-defined complete clips. Its design therefore targets efficient causal encoding, reusable context, and selective retrieval.

  • Task Definition: StreamingVQA continuously analyzes an incoming stream and answers questions using only frames seen by the query time.OfflineVQA instead operates on a predefined video and questions in an offline setting.
  • Discussion: Processing all frames with conventional visual encoders and language models is impractical because computational cost grows with video length.The standard pipeline concatenates encoded frames with tokenized questions before answer prediction.
  • Discussion: Sparse sampling loses critical visual information and requires reprocessing frames for questions asked at different time points.These inefficiencies increase as the numbers of frames and questions grow.
  • Design Principles: Causal masking decouples video encoding from question-answering because video tokens cannot access future tokens or the questions during encoding.This permits separate processing of the stream and subsequent queries.
  • Design Principles: Sliding-window attention limits video encoding to recent frames, while retrieved KV-Caches provide selected long-term context for answering questions.The retrieval stage avoids loading every past cache and supports scalable StreamingVQA.

3 REKV: RETRIEVE IN-CONTEXT VIDEO KV-CACHE

ReKV incrementally encodes streams with sliding-window attention, stores the resulting KV-Caches, retrieves question-relevant frames or blocks, and uses their caches for answer generation. Retrieval can be external or internal, with internal retrieval reusing Video-LLM representations across attention layers.

  • Framework: ReKV is a training-free Video-LLM framework comprising stream encoding, KV-Cache retrieval, and question-answering with retrieved vectors.It integrates with existing Video-LLMs without additional training.
  • Video Stream Encoding: Video chunks are encoded incrementally with sliding-window attention over a local window of recent past key-value vectors.All video KV-Caches are retained, with long-video caches offloaded to RAM or disk when needed.
  • External Retrieval: External retrieval embeds frames and questions with a CLIP-like model, scores cosine similarity, and loads the most relevant frame or block caches onto the GPU.Retrieval can operate on individual frames or blocks formed by averaging consecutive frame vectors.
  • Internal Retrieval: Internal retrieval represents frames using averaged key vectors and performs retrieval within Video-LLM self-attention layers.Different layers can retrieve different blocks while reusing hidden representations without extra parameters.
  • Question-Answering: Retrieved KV-Caches serve as context for autoregressive question-answering, with the context including retrieved video, question, and previously generated tokens.The positional-encoding treatment does not preserve the original positions of retrieved caches.

4 EXPERIMENTS

Experiments evaluate ReKV across long-form and streaming VideoQA benchmarks, retrieval settings, and efficiency measures. Results show that retrieval improves accuracy over uniform sampling, while internal retrieval supports strong performance with efficient streaming inference.

  • Benchmarks and Metrics: ReKV is evaluated on multiple-choice and open-ended long-form VideoQA benchmarks, including MLVUdev-mc, QAEGO4Dtest-mc, EgoSchema, and ActivityNet-QA.The evaluation also includes StreamingVQA benchmarks and experiments with LLaVA-OV-0.5B and LLaVA-OV-7B.
  • Retrieval Ablations: Higher retrieval recall consistently corresponds to better VideoQA accuracy on QAEGO4Dtest-mc, with Internal Retrieval outperforming Uniform Sampling but remaining below Oracle Retrieval.Oracle Retrieval uses annotated question-relevant segments and has 100% recall.
  • Retrieval Ablations: Both External and Internal Retrieval improve overall MLVU accuracy over Uniform Sampling, with the largest gains on Single Detail tasks.Internal Retrieval also significantly outperforms External Retrieval on Holistic tasks.
  • Retrieval Hyperparameters: Increasing retrieved frames generally improves accuracy, but gains plateau on MLVU as irrelevant information and question-answering computation increase.The experiment varies retrieved frames from 8 to 80 while fixing block size at b = 1.
  • Retrieval Hyperparameters: Larger retrieval blocks reduce MLVU accuracy while leaving QAEGO4D performance relatively stable, indicating that MLVU benefits from more dispersed visual cues.The experiment fixes retrieved frames at r = 64 and varies block size from 1 to 16.
  • Overall and Streaming Results: ReKV improves both LLaVA-OV-0.5B and LLaVA-OV-7B without additional training and enables LLaVA-OV-7B to outperform memory-based StreamingVQA models.In StreamingVQA, both retrieval methods outperform Uniform Sampling, while LLaVA-OV-7B achieves 11 FPS and LLaVA-OV-0.5B achieves 17 FPS.

5 RELATED WORK

Related work addresses long-video understanding through language compression, memory representations, retrieval, and long-context attention. ReKV instead preserves video KV-Caches and retrieves relevant information for training-free StreamingVQA.

  • Video-LLMs typically combine vision-encoder features with mapped textual inputs for video understanding.
  • Language-based video compression can lose crucial visual details, while memory-bank methods condense lengthy videos into stored representations.
  • Unlike prior memory-based approaches requiring fine-tuning, ReKV is training-free and tailored specifically to StreamingVQA.
  • ReKV preserves the entire Video KV-Cache and retrieves relevant information instead of excessively condensing video content.
  • Sliding-window attention processes long sequences incrementally but discards distant tokens, limiting long-range dependency modeling.
  • ReKV applies in-context retrieval by selecting information from the input video rather than relying on external knowledge.

6 CONCLUSION

The paper concludes that ReKV is a training-free framework for efficient StreamingVQA. It combines streaming encoding, in-context KV-Cache retrieval, and separated processing to improve performance and practicality.

  • ReKV enables rapid, real-time responses to questions about long video streams without additional training.
  • Sliding-window attention reduces encoding computation by restricting attention to a subset of previous frames.
  • In-context KV-Cache retrieval stores and reloads query-relevant key-value vectors for answering questions efficiently.
  • Separating video modeling and question-answering across processes and GPUs improves the practicality of streaming VideoQA.
  • Extensive experiments report that ReKV surpasses existing VideoQA models in performance and enhances applicability to real-world streaming.

A ADDITIONAL IMPLEMENTATION DETAILS

The implementation separates video encoding from question-answering and manages KV-Caches across GPU memory, RAM, and disk. The section also specifies prompt formats and cache-size calculations.

  • Implementation architecture: A primary encoding process uses sliding-window attention and stores computed video caches in RAM, offloading data to disk when RAM is exceeded.
  • Prompt templates: Multiple-choice VideoQA uses a fixed prompt containing the video, question, five answer options, and an instruction to return the option letter.
  • Prompt templates: Open-ended VideoQA uses a simpler prompt containing the video and question before the assistant response.
  • KV-Cache size: 2 × L layers × T frames × M tokens/frame × H heads × D dimension × 2 bytes gives the FP16 KV-Cache size.
  • KV-Cache size: 18.8 GB is the calculated KV-Cache size for LLaVA-OV-7B processing a 1-hour video at 0.5 FPS.
  • KV-Cache size: 4.0 GB is the corresponding KV-Cache size for LLaVA-OV-0.5B under the same 1-hour, 0.5 FPS setting.
  • KV-Cache size: The theoretical cache-size calculations match the experimental results reported in Table 5.

B.1 EXPERIMENTS WITH MORE VIDEO-LLMS AND BENCHMARK

Additional experiments evaluate ReKV with Video-LLaVA-7B, LongVA-7B, and LLaVA-OV-72B under adjusted evaluation settings. ReKV consistently improves performance across models and benchmarks.

  • ReKV was tested on Video-LLaVA-7B, LongVA-7B, and LLaVA-OV-72B to assess generalizability.
  • For LLaVA-OV-72B, model sharding slowed inference, so evaluation used 0.1 FPS and 32 retrieved frames.
  • ReKV consistently improved performance across the additional Video-LLMs and benchmarks.Table 6 reports the additional-model and benchmark experiments; “Acc.” denotes accuracy and frame notation describes sampling or retrieval settings.

B.2 FAIR COMPARISONS WITH FLASH-VSTREAM

The fair comparison uses an identical Video-LLM backbone and controlled training conditions to compare Base, Base+Flash, and Base+ReKV. Under these conditions, Base+ReKV consistently outperforms Base and surpasses Base+Flash in most cases.

  • Controlled comparison: Fair comparisons use the same Video-LLM backbone, visual encoder, projector, LLM, training data, and train/evaluation pipelines.The backbone uses CLIP-ViT-L/14, a 2-layer MLP, and Vicuna-7B-v1.5.
  • Controlled comparison: The study trains and compares Base, Base+Flash, and Base+ReKV under matched experimental settings.InternVid videos substitute for inaccessible WebVid videos, while the baseline uniformly samples 16 frames resized to 224 × 224.
  • Results: Base+ReKV consistently outperforms Base and surpasses Base+Flash in most cases under fair comparative conditions.The table compares the reproduced Base+Flash version with the same-backbone Base and Base+ReKV systems.
  • Results: Base+Flash excels on StreamingVQA and MLVU but underperforms on QAEGO4D and EgoSchema relative to Base.The discrepancy is attributed to visual-information loss: Base processes 1024 visual tokens, whereas Base+Flash uses 681 memory tokens.
  • Results: The reproduced Base+Flash differs from the original Flash-VStream checkpoint, likely because of training-data and environmental differences.The original checkpoint was obtained from Flash-VStream’s official repository.

B.3 COMPUTATIONAL COMPLEXITY

The complexity analysis measures FLOPs and MACs per question under controlled streaming conditions and varying question frequencies. ReKV becomes more efficient as question frequency rises, while internal retrieval reduces complexity relative to external retrieval.

  • Measurement setup: The analysis measures average TFLOPs and TMACs per question for a 1-hour video across question frequencies.It compares the base Video-LLM, Flash-VStream, and ReKV’s external and internal retrieval methods using identical backbones and controlled streaming conditions.
  • Frequency scaling: ReKV’s efficiency improves significantly as QA frequency increases because the video stream is encoded once and reused across questions.Reusing computed results reduces per-query complexity when more questions are asked about the same stream.
  • Frequency scaling: Flash-VStream outperforms ReKV at low QA frequencies, such as 100 QAs, but ReKV’s complexity decreases more rapidly as frequency rises.Flash-VStream’s high memory-update overhead primarily explains the different scaling behavior.
  • Implications: ReKV is better suited for high-concurrency scenarios such as live streaming and requires no additional training.This conclusion follows the reported frequency-scaling analysis and the framework’s training-free design.
  • Retrieval comparison: Internal retrieval reduces average FLOPs by 15.5% and MACs by 15.2% compared with external retrieval.Internal retrieval consistently outperforms external retrieval in the reported efficiency comparison.

C LIMITATIONS AND FUTURE WORK

The paper identifies several limitations and future directions, including cache growth for extremely long streams, fixed retrieval blocks and frame counts, and limited StreamingVQA benchmarks.

  • Cache scalability: Handling extremely long video streams, such as surveillance footage, may cause an unsustainable increase in KV-Cache size.The paper suggests quantization, token pruning, and compression as possible mitigations.
  • Retrieval granularity: Using a constant block size to group consecutive frames during retrieval can disrupt video continuity.Future work could segment videos into semantically coherent blocks.
  • Retrieval adaptivity: Retrieving a fixed number of frames does not adapt to video context or query requirements.The paper proposes dynamic retrieval strategies that adjust the number of frames.
  • Evaluation resources: StreamingVQA has few available benchmarks, motivating datasets with precise temporal annotations.The paper identifies high-quality benchmark development as important for advancing future research.
Loading 2503.00540v1…