Source-linked AI summary
Streaming Long Video Understanding with Large Language Models
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, Jiaqi Wang
TL;DR
Long-video understanding requires many tokens, while existing sparse or frame-wise compression can lose temporal or spatial information. VideoStreaming propagates memory across sequentially encoded clips and adaptively selects question-related memories for LLM reasoning. The model reports superior performance, precise temporal comprehension, and higher efficiency on long-video benchmarks, with uniform clip sampling as a stated limitation.
Problem
Long videos impose substantial token and computation demands, while sparse sampling or frame compression can discard long-range temporal dynamics or spatial detail.
Method
VideoStreaming sequentially encodes short clips with propagated historical memory, stores the resulting memories, and adaptively selects a constant number of question-related memories for the LLM.
Results
VideoStreaming achieves superior performance and higher inference efficiency on long-video benchmarks, including consistent gains over zero-shot counterparts on Next-QA.
Takeaways & Limitations
The framework supports detailed long-video question answering by combining fixed-length streaming representations with question-specific historical memory selection.
Takeaways & Limitations
Uniformly sampled clips may waste tokens on bland segments and provide insufficient tokens for visually dense, dynamically changing segments.
Abstract
from arXiv · showhide
This paper presents VideoStreaming, an advanced vision-language large model (VLLM) for video understanding, that capably understands arbitrary-length video with a constant number of video tokens streamingly encoded and adaptively selected. The challenge of video understanding in the vision language area mainly lies in the significant computational burden caused by the great number of tokens extracted from long videos. Previous works rely on sparse sampling or frame compression to reduce tokens. However, such approaches either disregard temporal information in a long time span or sacrifice spatial details, resulting in flawed compression. To address these limitations, our VideoStreaming has two core designs: Memory-Propagated Streaming Encoding and Adaptive Memory Selection. The Memory-Propagated Streaming Encoding architecture segments long videos into short clips and sequentially encodes each clip with a propagated memory. In each iteration, we utilize the encoded results of the preceding clip as historical memory, which is integrated with the current clip to distill a condensed representation that encapsulates the video content up to the current timestamp. After the encoding process, the Adaptive Memory Selection strategy selects a constant number of question-related memories from all the historical memories and feeds them into the LLM to generate informative responses. The question-related selection reduces redundancy within the memories, enabling efficient and precise video understanding. Meanwhile, the disentangled video extraction and reasoning design allows the LLM to answer different questions about a video by directly selecting corresponding memories, without the need to encode the whole video for each question. Our model achieves superior performance and higher efficiency on long video benchmarks, showcasing precise temporal comprehension for detailed question answering.
1 Introduction
Long-video understanding burdens LLMs with many tokens and can lose temporal or spatial information under common compression strategies. VideoStreaming addresses this with streaming memory propagation and question-driven memory selection.
- Long visual contents require many tokens, increasing computation and risking loss of early contextual information.
- Sparse sampling and spatio-temporal pooling reduce tokens but lose information across long time spans.
- Frame-wise compression preserves fewer tokens per frame but overlooks inter-frame temporal dynamics needed to compress temporal redundancy.
- VideoStreaming sequentially encodes short clips using preceding memories, producing fixed-length representations that capture information up to each timestamp.A small decoder-only language model integrates historical memory with current clip features, and its final tokens become updated memory.
- Adaptive Memory Selection stores all historical memories and selects a constant number of question-related subsets to recover details lost by fixed-length memory.Summary tokens act as clip indicators, while selected memories are passed to the LLM for question answering.
- A two-stage progressive training process equips the encoder for single-clip encoding before jointly training it with the LLM for long-video understanding.The authors also construct long-video question-answering data because existing long-video QA data is limited.
- Experiments report precise temporal grounding, superior long-video benchmark performance, and higher inference efficiency.
2 Related Work
Related work applies LLMs and vision-language models to visual reasoning, while long-video systems compress or caption clips to manage memory. VideoStreaming instead uses trainable iterative encoding of clips into compact memories.
- Decoder-only LLMs such as GPT demonstrate scalability, and instruction tuning guides them toward natural, contextually relevant responses.
- Video-language methods extend visual-language modeling to spatio-temporal data using sparse sampling or simple temporal processing.
- Long-video systems commonly maintain memory banks and compress history into finite-length memories using parametric or non-parametric modules.
- Language-bridging approaches caption short clips before LLM aggregation, but their quality depends on captions and they cannot be trained end-to-end.
- VideoStreaming uses a trainable small language model to iteratively encode clips into compact memories jointly optimized with the downstream LLM.
3 VideoStreaming
VideoStreaming encodes long videos clip by clip into fixed-length historical memories, then selects question-relevant memories for LLM reasoning. Its design combines temporal context propagation with adaptive retrieval to preserve detail while maintaining efficient video understanding.
- 3. VideoStreaming: VideoStreaming segments long videos into short clips and iteratively encodes each clip into compact historical memory for subsequent processing.The streaming encoder uses preceding encoded results together with current clip features and summarization tokens.
- 3.1 Single Clip Encoding: The single-clip encoder extracts CLIP frame features, merges adjacent spatial tokens, and uses Phi-2 with summarization tokens to produce condensed representations.Concatenating every four adjacent visual tokens reduces the number of tokens by 75%, and the last T × P outputs form the clip representation.
- 3.1 Single Clip Encoding: A prefix training task modifies decoder attention so response tokens access video information through summarization tokens, encouraging visual consolidation.The task uses clip feature tokens, summarization tokens, and response tokens in a constrained attention structure.
- 3.2 Memory-Propagated Streaming Long Video Encoding: Each updated memory represents the current clip and the video content accumulated through the corresponding timestamp while keeping memory length fixed.For the first clip, no historical memory is used; later iterations propagate the preceding memory.
- 3.3 Adaptive Memory Selection: Because final global memory can lose early-segment details, adaptive selection retrieves fine-grained information from historical memories while preserving computational efficiency.The model separates video extraction from reasoning, allowing relevant memories to be selected for specific instructions.
- 3.3 Adaptive Memory Selection: Adaptive memory selection compares an instruction indicator with historical clip indicators and activates a question-related subset using Gumbel-Topk.Selected memories are concatenated in temporal order and passed with the instruction to an LLM, producing V × T × P tokens.
4 Experiments
Experiments evaluate VideoStreaming across minute-long, three-minute, ten-minute, and hour-long video benchmarks, covering question answering, temporal grounding, and efficiency. The model reports strong performance, question-related timestamp selection, and benefits from historical memory and adaptive selection.
- 4.1 Datasets: The evaluation spans long-video datasets from minute-long clips to hour-long movies, including QA, multiple-choice, temporal grounding, and movie-understanding tasks.EgoSchema contains over 5K three-minute videos, while MovieNet-QA contains 100 hour-long movies.
- 4.2 Main Results: VideoStreaming outperforms LLM-based video understanding methods on all five VideoChatGPT metrics, with a significant advantage in temporal understanding.The reported metrics are Correctness of Information, Detailed Orientation, Contextual Understanding, Temporal Understanding, and Consistency.
- 4.2 Main Results: VideoStreaming outperforms all zero-shot EgoSchema methods and is comparable to finetuned MC-ViT for long-term temporal modeling.The comparison contrasts trainable streaming memory embeddings with methods based on captions from segmented clips.
- 4.2 Main Results: 0.7%, 10.8%, and 9.0% accuracy improvements are reported on Next-QA causal, temporal, and descriptive subsets versus LangRepo with Mixtral-8×7B.The comparison uses the authors’ 8.3B model and reports considerably fewer model parameters.
- 4.2 Main Results: The model achieves the highest IoP and highest Acc@GQA on Next-GQA, with IoU comparable to SeViLA.Acc@GQA measures questions that are both correctly answered and visually grounded with IoP ≥0.5.
- 4.2 Main Results: 5.32 seconds per question is the reported MovieNet-QA inference speed, compared with over 10 seconds for approaches that reprocess the movie per question.VideoStreaming performs one streaming encoding, then selects fewer tokens for question-specific LLM reasoning without subtitles.
- 4.3 Ablation Study: 46.6% higher global understanding results from historical memory, while removing temporal memory selection causes a 31.9% breakpoint-performance drop.Historical memory supports global summarization, whereas adaptive selection helps detailed questions about specific moments.
5 Conclusion
VideoStreaming combines sequential memory-propagated encoding with adaptive selection to represent arbitrarily long videos using fixed-length memories and answer timestamp-specific questions efficiently.
- Memory-propagated streaming encoding segments long videos into short clips and iteratively incorporates preceding memory to capture temporal dynamics in a fixed-length representation.
- Adaptive memory selection chooses question-relevant historical timestamps, supplying pertinent memories for detailed question answering.
- VideoStreaming achieves superior performance with substantially fewer tokens and higher efficiency on extensive long-video benchmarks.
- Streaming memories enhance global video understanding, while adaptive selection supports accurate temporal grounding for specific questions.
Limitations
The method uniformly samples frames into short clips, which may allocate tokens inefficiently because information density varies across long videos.
- Uniform frame sampling can waste tokens on bland clips while providing too few tokens for visually rich clips with intensive temporal dynamics.The resulting imbalance may cause information loss in content-dense segments.
- The authors identify adaptive segmentation, with clip lengths adjusted to video complexity and content, as a direction for addressing this limitation.
Impact Statements
VideoStreaming may improve long-video understanding for educational, entertainment, retrieval, and multimedia-analytics applications, but its capabilities also raise privacy, security, and misinformation concerns.
- VideoStreaming could benefit education, entertainment, and information retrieval by improving understanding of complex long videos.
- The technique could advance multimedia analytics in video surveillance, market research, and content personalization.
- Efficient long-video information retrieval raises privacy and security concerns, including possible unauthorized surveillance and personal monitoring.
- Enhanced video understanding could be exploited to create misleading video content, potentially contributing to misinformation and social manipulation.
- The paper concludes that developing VideoStreaming should include careful consideration of ethical and societal implications.
A More Implementation Details
The implementation uses CLIP frame features, a Phi-2-based streaming encoder, timestamp prompts, adaptive timestamp selection, and Vicuna-based response generation, with long-video data assembled from multiple sources.
- CLIP ViT-L/14 extracts 256 tokens per 224×224 frame, which are grouped into 64 tokens per frame with channel dimension 4096.
- The streaming encoder combines a two-layer MLP projector with GELU activation and the Phi-2 2.7B language model.
- Timestamp prompts explicitly identify historical-memory and current-clip intervals during streaming encoding.
- The system selects 4 relevant timestamps and feeds their 256 memory tokens through an MLP projector and Vicuna-7B to generate responses.
- Panda-70M provides caption data with timestamped descriptions such as a reporter, people outside a building, and children on a television screen.
- Table 11 reports an ablation study varying summarization-token counts and selected timestamps on EgoSchema and Next-GQA.
- The full training process uses next-token prediction loss on 32 A100 80G GPUs for approximately 2.5 days.
B Long Video QA Data Creation
The paper expands long-video QA data by synthesizing minute-long videos from short-video QA datasets and using temporal correspondences as noisy memory-selection supervision.
- Minute-long videos are synthesized from existing short-video QA datasets, with original questions coarsely aligned to temporal segments.These correspondences provide noisy labels for supervising memory selection.
C More Ablation Studies
The ablations examine summarization capacity, timestamp prompts, and similarity measurement, showing that adequate spatial detail, temporal cues, and stable similarity computation are important for performance.
- The Number of Summarization Tokens and Selected Timestamps: Too few summarization tokens, such as P = 1, substantially reduce performance by compressing 16 frames into only 16 tokens and losing spatial information.
- Time Prompts: Missing time prompts substantially harms MovieNet-1K breakpoint accuracy, which requires detailed reasoning about specified moments.
- Time Prompts: Including historical-memory timestamps improves global understanding, while jointly using historical-memory and current-clip timestamps produces the best overall results.
- Similarity Measurement: Dot-product similarity can become numerically unstable and overflow during training, producing poor results because scores no longer reflect instruction-segment correlation.