Source-linked AI summary
VideoLLM-online: Online Video Large Language Model for Streaming Video
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, Mike Zheng Shou
TL;DR
Existing multimodal video methods generally assume predetermined clips, leaving continuous streaming dialogue insufficiently supported for temporal alignment, long context, and real-time interaction. The paper introduces LIVE, combining streaming-specific learning, data generation, and inference, and builds VideoLLM-online on Llama-2/Llama-3. VideoLLM-online supports 5-minute streaming dialogue at over 10 FPS with less than 20 GB memory on one A100 GPU and achieves strong offline benchmark performance.
Problem
Existing video multimodal models typically process predetermined clips, while continuous streaming dialogue requires temporal alignment, long-context handling, and real-time responses.
Method
LIVE combines Streaming EOS training, conversion of temporal annotations into streaming dialogue data, and efficient inference with continuous key-value caching and parallel visual encoding.
Results
VideoLLM-online supports 5-minute streaming dialogue at over 10 FPS with less than 20 GB memory on one A100 GPU and achieves state-of-the-art results on numerous offline benchmarks.
Takeaways & Limitations
LIVE advances VideoLLM-online toward always-on video assistance by combining temporally aligned responses, long-context streaming, and efficient inference.
Takeaways & Limitations
The proposed streaming narration metrics are not effective for evaluating more complicated free-form online conversation scenarios.
Abstract
from arXiv · showhide
Recent Large Language Models have been enhanced with vision capabilities, enabling them to comprehend images, videos, and interleaved vision-language content. However, the learning methods of these large multimodal models typically treat videos as predetermined clips, making them less effective and efficient at handling streaming video inputs. In this paper, we propose a novel Learning-In-Video-Stream (LIVE) framework, which enables temporally aligned, long-context, and real-time conversation within a continuous video stream. Our LIVE framework comprises comprehensive approaches to achieve video streaming dialogue, encompassing: (1) a training objective designed to perform language modeling for continuous streaming inputs, (2) a data generation scheme that converts offline temporal annotations into a streaming dialogue format, and (3) an optimized inference pipeline to speed up the model responses in real-world video streams. With our LIVE framework, we built VideoLLM-online model upon Llama-2/Llama-3 and demonstrate its significant advantages in processing streaming videos. For instance, on average, our model can support streaming dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU. Moreover, it also showcases state-of-the-art performance on public offline video benchmarks, such as recognition, captioning, and forecasting. The code, model, data, and demo have been made available at https://showlab.github.io/videollm-online.
1. Introduction
Existing multimodal models largely process manually selected video clips rather than continuous streams, which require temporal alignment, long-context handling, and real-time responses. LIVE addresses these requirements through streaming-specific learning, data generation, and inference methods, while VideoLLM-online achieves strong streaming and offline performance.
- Motivation: Existing VideoLLMs typically operate offline on manually selected short clips, unlike assistants that continuously receive refreshed video frames.Streaming dialogue must respond to temporally aligned requests and inspect incoming frames to avoid missing events.
- Motivation: Continuous video assistance must determine when to answer, preserve long context, and generate responses fast enough to keep pace with the stream.Long streams can exceed language-model context windows and increase decoding speed and GPU-memory burdens.
- Challenges: Per-frame chatting with GPT-4V produces lengthy responses and delays, while baseline per-frame training can diminish language-modeling capability.The latter is attributed to harmful language modeling on excessive redundant frames.
- LIVE Framework: LIVE combines a streaming training objective, streaming-dialogue data generation, and an optimized inference pipeline for online video assistance.Its Streaming EOS objective teaches the model when to respond or remain silent and reduces unnecessary context.
- Results: VideoLLM-online supports 5-minute streaming dialogue at over 10 FPS with less than 20 GB memory on one A100 GPU.The model also achieves state-of-the-art results on several offline recognition and forecasting benchmarks.
2. Related Work
Prior multimodal research developed image dialogue, interleaved vision-language interaction, spatial understanding, and offline video comprehension. Related work leaves online streaming video's temporal-alignment, long-context, and real-time requirements less explored.
- Visual Dialogue: Early visual-dialogue systems used visually enhanced encoders with discriminative answer selection or recurrent architectures for multi-turn responses.Later approaches also explored attention mechanisms and encoder-only BERT models.
- Large Multimodal Models: Large multimodal models expanded from image dialogue to interleaved vision-text chatting, spatial understanding, and video comprehension.These systems commonly project visual encodings, such as CLIP features, into the language model's embedding space.
- Online Video Understanding: Online streaming video remains less explored than offline benchmarks that let models access entire videos before prediction.The paper frames its contribution as addressing temporal alignment, long context, and real-time requirements across training, data, and inference.
3. Method
LIVE frames video streaming dialogue as continuous video-language interaction requiring temporal response alignment, long-context handling, and real-time processing. It addresses these requirements through streaming-aware training, generated dialogue data, and an optimized inference pipeline.
- Problem definition: Video streaming dialogue requires deciding when to respond while continuously processing frames and carrying forward prior vision-language context.The formulation includes historical queries, frames, and responses before the stream, followed by ongoing frames and language modeling at suitable timestamps.
- Challenges: Existing interleaved or per-frame dialogue approaches require manually selected timestamps or repeatedly perform costly language modeling, limiting streaming alignment and efficiency.Per-frame processing can cause frame skipping, consume context, and increase GPU-memory demands.
- Streaming EOS prediction: Streaming EOS prediction trains the model to emit EOS on redundant frames, allowing it to skip dialogue turns and decode language only at appropriate timestamps.The EOS token is not appended to context during inference, preventing it from affecting language modeling.
- Streaming dialogue data: The data-generation scheme inserts templated queries at random video timestamps and uses timestamped annotations with LLM-generated responses to create temporally varied streaming dialogue.Training samples can contain multiple inserted queries, with a maximum of 3 queries per sample and N = 150 queries spanning past, current, and future events.
- Inference: The inference pipeline maintains a continuous key-value cache and parallelizes frame encoding with language modeling so frames can be encoded and buffered without waiting for decoding.This design supports continuous frame-by-frame processing and targets faster real-time streaming responses.
- Model architecture: VideoLLM-online combines a CLIP ViT-L encoder, MLP projector, and Llama-2 or Llama-3 language model, with frame embeddings extracted at 2 FPS and LoRA-based tuning.Experiments use an efficient setup without extra spatial tokens, while released demo models use 10 tokens per frame for greater detail and still run over 10 FPS on 5-minute streams.
4. Experiments
Experiments evaluate VideoLLM-online on streaming narration, offline video benchmarks, model variants, and qualitative streaming dialogue. The results show improved streaming metrics, competitive offline performance, and efficient real-time behavior.
- Model Setup: VideoLLM-online uses CLIP-based frame encoding with Llama-2/Llama-3 language models, including efficient and more capable variants.The 7B variant uses one CLS token per frame, while the 8B-v1+ variant uses ten tokens per frame.
- Evaluation Setting: Streaming narration experiments use Ego4D-derived datasets, with additional COIN and Ego4D benchmarks covering recognition, summarization, and forecasting.The evaluation includes Ego4D Narration Stream, COIN+Ego4D Narration Stream, six COIN benchmarks, and Ego4D long-term action anticipation.
- Ablation Study: Streaming dialogue yields much better TimeDiff and Fluency than interleaved and per-frame alternatives, while avoiding the latter’s lengthy prompts and training inefficiency.Per-frame learning also produces significantly higher LM-PPL, whereas the proposed streaming objective preserves language-modeling capability.
- Offline Results: VideoLLM-online achieves state-of-the-art performance on COIN step/task summarization and next-step/procedure forecasting, and the best result among end-to-end Ego4D LTA models.AntGPT scores higher but uses egocentric pre-trained features and complex cascading methods; VideoLLM-online directly generates language.
- Model Variants: The enhanced language model improves performance across evaluated aspects, while multiple tokens per frame strengthen vision-language capability but provide limited online gains.The comparison covers VideoLLM-online-7B-v1, VideoLLM-online-8B-v1, and VideoLLM-online-8B-v1+.
- Visualization: Qualitative examples show responses aligned with streaming frames and muted when unnecessary, supporting faster online dialogue.The inference strategy is presented as enabling real-time assistance rather than full-video offline chatting.
5. Conclusion
The paper presents LIVE as a framework for temporally aligned, long-context, efficient streaming video understanding. VideoLLM-online improves online and offline vision-language capabilities, while future work targets broader generality and spatial capability.
- Conclusion: LIVE enables temporally aligned answers, long-context video handling, and high inference efficiency for streaming video.The framework is used to train the VideoLLM-online model.
- Conclusion: VideoLLM-online achieves superior capability across online and offline vision-language tasks and supports fast inference in online video streaming settings.The authors position these abilities as an important step toward always-on online assistants.
- Future Work: Future work will seek more general models with stronger zero-shot spatial capability and more spatial tokens without large speed or memory trade-offs.The paper also identifies suitable pre-training data sources as a future direction.
Supplementary Material
The supplementary material analyzes per-frame chatting, documents data and evaluation details, reports additional stream-set results, presents demos, and discusses limitations.
- A. Analysis to Per-frame Chatting: Section A analyzes GPT-4V per-frame chatting against interleaved vision-language dialogue and VideoLLM-online.
- B. Data and Evaluation Details: Section B details prompts, streaming datasets, training and inference formats, and evaluation schemes for COIN and Ego4D benchmarks.
- Additional Results and Demos: The supplementary material includes Ego4D and COIN stream-set results and VideoLLM-online demonstrations.
- D. Limitations: A dedicated section discusses limitations of the paper.
A. Analysis to Per-frame Chatting
GPT-4V’s per-frame chatting is impractical for real-time narration because unrestricted responses are lengthy, while strong prompting remains costly and unstable.
- Unrestricted Prompting: GPT-4V produces very lengthy responses at every frame without prompting restrictions, making real-time usage impractical.
- Strong Prompting: Strong prompting can make GPT-4V approach video streaming dialogue, but it remains per-frame dialogue that consumes tokens and time at every frame.
- Strong Prompting: Strongly prompted GPT-4V is unstable and can hallucinate contradictory actions across adjacent frames.The passage contrasts “you are peeling” with “you have stopped” at the first and second frame.
B.1. Data Construction
The supplementary material constructs streaming dialogue data and prompts for COIN and Ego4D, using frame tokens to represent continuously observed video in training and inference.
- COIN Stream Set: COIN Stream Set adapts COIN annotations into streaming dialogue, with an initial task query followed by real-time activity responses.
- COIN Stream Set: COIN Stream Set uses a fixed beginning-only query for stable evaluation, whereas demo models use randomized queries, timestamps, and turn counts.
- Ego4D Narration Stream Set: Ego4D Narration annotations inherently follow a streaming dialogue format, and videos are segmented into clips of at most five minutes.
- Training and Inference Prompt: Each video frame is encoded with frozen CLIP ViT and projected through a learnable MLP into language-token space.
- Training and Inference Prompt: Experiments use one token per frame, while demonstrations use ten tokens per frame consisting of one CLS token and 3 × 3 average-pooled spatial tokens.
- Streaming Dialogue Examples: Streaming examples include queries that request reminders when actions start, summaries when they end, and forecasts of subsequent actions.
B.3. Evaluation Scheme
The evaluation scheme converts VideoLLM-online’s generated text into benchmark-compatible outputs for COIN and Ego4D, while supplementary figures and tables document demos and comparative results.
- COIN Benchmarks: COIN benchmarks report top-1 accuracy after string matching generated language against the COIN taxonomy dictionary.
- COIN Benchmarks: For multi-step COIN procedures, semicolon-separated outputs are split into individual steps for step-wise accuracy calculation.
- Demos and Comparative Results: Figure 7 presents an online narration demo, Figure 8 presents an online chatting demo, and Table 4 compares joint stream training with per-frame dialogue.
- Ego4D LTA: Ego4D LTA uses Edit Distance, measuring the minimum operations needed to transform one string into another.
- Evaluation Limitation: The COIN evaluation method is limited to single-step or overall-procedure outputs rather than more complex text outputs.
- Ego4D LTA: Generated text is converted to verb/noun category indices through string splitting, taxonomy matching, padding or clamping, and dictionary lookup.
C. More Results
On joint COIN and Ego4D streaming sets, the method matches per-frame dialogue in language modeling while substantially improving fluency and temporal alignment.
- Similar LM-PPL to per-frame video-language dialogue, while achieving huge advantages in fluency and time difference.
D. Limitations
The primary limitation is insufficient high-quality streaming dialogue data, which constrains generalization and can cause overfitting on small datasets.
- Insufficient high-quality streaming dialogue data limits the method’s generalization capability.
- Generated dialogues derive from existing video datasets and cannot capture complex, varied real-world user requirements.
- Training on a small dataset can cause overfitting.
- The model’s spatial ability is weak because it uses fewer spatial tokens, motivating a better spatial-temporal trade-off.