PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamChat: Chatting with Streaming Video

Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, Jose M. Alvare

arXiv:2412.08646v2cs.CV

TL;DR

Streaming models that rely only on video available when a question is asked can miss changes while decoding a response. StreamChat updates visual context at each decoding step and combines cross-attention, streaming-oriented instruction data, and temporal encoding. It is competitive on established image and video benchmarks and shows stronger streaming interaction, including outperforming LLaVA-Video-72B with StreamChat-7B.

  • Problem

    Existing methods typically use video only up to the moment a question is asked, leaving them unaware of subsequent changes during answer decoding.

    Source: §1 · p. 2

  • Method

    StreamChat updates visual context at each decoding step and combines cross-attention with dense instruction data and parallel 3D-RoPE for streaming interaction.

    Source: §1 · p. 2

  • Results

    StreamChat achieves competitive image and video benchmark performance, improves streaming interaction, and StreamChat-7B outperforms LLaVA-Video-72B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    StreamChat’s visual context updates support responses aligned with dynamic changes in streaming video.

    Source: §8 · p. 9

    Word timestamps are generated heuristically from coarse temporal annotations, which may reduce alignment accuracy in complex video scenarios requiring precise timing.

    Source: §9 · p. 9

Abstract

from arXiv · show

This paper presents StreamChat, a novel approach that enhances the interaction capabilities of Large Multimodal Models (LMMs) with streaming video content. In streaming interaction scenarios, existing methods rely solely on visual information available at the moment a question is posed, resulting in significant delays as the model remains unaware of subsequent changes in the streaming video. StreamChat addresses this limitation by innovatively updating the visual context at each decoding step, ensuring that the model utilizes up-to-date video content throughout the decoding process. Additionally, we introduce a flexible and efficient crossattention-based architecture to process dynamic streaming inputs while maintaining inference efficiency for streaming interactions. Furthermore, we construct a new dense instruction dataset to facilitate the training of streaming interaction models, complemented by a parallel 3D-RoPE mechanism that encodes the relative temporal information of visual and text tokens. Experimental results demonstrate that StreamChat achieves competitive performance on established image and video benchmarks and exhibits superior capabilities in streaming interaction scenarios compared to state-of-the-art video LMM.

1. Introduction

StreamChat targets streaming-video interaction, where models that use only video available when a question is asked can miss changes during decoding. It updates visual context as it generates answers, with a streaming-oriented architecture, training data, and temporal encoding.

1. Introduction

  • Existing methods use video only up to the question time, missing changes during answer decoding and impairing interaction, especially in dynamic videos or with lengthy answers.If answering takes t′ seconds after a question at t, these methods do not use changes from t to t + t′.Source: §1 · pp. 1–2
  • StreamChat updates visual context at each decoding step and uses cross-attention to handle variable-length streaming inputs efficiently, particularly with many visual tokens.The architecture bridges language and visual inputs with cross-attention.Source: §1 · p. 2
  • The new dense instruction data simulates streaming by timestamping instruction-answer words and restricting each text token to video information available before its timestamp.The data is based on existing dense-caption datasets.Source: §1 · p. 2
  • Parallel 3D-RoPE encodes temporal, height, and width positions while giving visual and text tokens at the same timestamp a shared temporal position.Absolute timestamps are not input because they are unavailable during inference.Source: §1 · p. 2
  • StreamChat achieves competitive image and video benchmark performance and improves streaming interaction; StreamChat-7B outperforms LLaVA-Video-72B.The paper also reports significantly improved streaming interaction capabilities.Source: §1 · p. 2

2. Methods

The methods combine a streaming-capable architecture with training data and inference procedures designed around changing video context. Visual tokens are updated as text is generated, while temporal alignment is represented in the model and training inputs.

2.1. StreamChat Architecture

  • StreamChat’s architecture combines cross-attention for visual-text interaction, V-FFN experts for visual-token updates, and parallel 3D-RoPE for temporal information.The overview presents these components as responses to the dynamic inputs of streaming video.Source: §2.1 · p. 3
  • Text tokens query visual tokens through cross-attention, while V-FFN experts update visual representations between blocks instead of reusing identical representations throughout.The V-FFN experts are initialized from the LLM’s FFN; a linear gate scales cross-attention and V-FFN outputs during initial training.Source: §2.1 · p. 3

2.2. Dense Instruction Data

  • Existing offline video instruction data does not match streaming access, so the method constructs a video instruction-tuning dataset from timestamped dense captions.Each streaming text token should access only part of the video, including frames at or before its timestamp.Source: §2.2 · p. 4

2.3. Training and Inference

  • Heuristic word timestamps and attention masks limit each token to video frames available at its time; at inference, a FIFO queue supplies the latest extracted visual tokens for each decoding step.The timestamps serve as references rather than direct model inputs, and the attention mask blocks access to future frames.Source: §2.3 · p. 4

3. Experiment Setups

The experiments use SigLIP with PaliGemma weights and Qwen 2.5 language models, followed by pretraining and instruction tuning. The supplied evaluation captions report stronger streaming interaction for StreamChat than comparison models.

3. Experiment Setups

  • The models use a SigLIP vision encoder with PaliGemma weights, 7B or 14B Qwen 2.5 language models, and an MLP adapter to align component dimensions.Source: §3 · p. 4

3.1. Pretraining

  • Pretraining proceeds in two stages: first the MLP adapter is trained, then the vision encoder and V-FFN experts are unfrozen for deeper alignment.The two stages each use 5,000 steps, with a maximum learning rate of 5 × 10−4 and 2 × 10−5, respectively.Source: §3.1 · p. 4

3.2. Instruction Tuning

  • Instruction tuning combines Eagle-1.8M, the dense instruction dataset, and LLaVA-Video for one epoch, using 2.9 million samples.Source: §3.2 · pp. 4–5
  • On streaming evaluation, StreamChat models show stronger interaction than VILA and LLaVA-Video, while examples show adaptation to video changes between questions at 1 and 21 seconds.The figure caption reports that StreamChat can outperform LLaVA-Video-72B; the qualitative caption contrasts its answers with factual errors from VILA and LLaVA-Video.Source: Figure 5 · p. 5, Figure 6 · p. 5

4. Streaming Evaluation

StreamChat is evaluated on a streaming-interaction benchmark built from timestamped video instructions and judged through pairwise answer comparisons. It outperforms competing models in quantitative comparisons and better maintains temporal alignment in a qualitative example.

4. Streaming Evaluation

  • The benchmark contains 100 manually reviewed samples: 80 from Vript and 20 from Ego4D, with each instruction-answer pair tied to a specific timestamp.Samples are generated from dense captions using Gemini-1.5-Pro, with samples unrelated to streaming scenarios removed.Source: §4 · pp. 5–6
  • Gemini-1.5-Pro judges paired model answers for adherence, helpfulness, relevance, and accuracy, then reports their overall win rate against StreamChat.The judge receives the ground-truth answer and both model outputs and may declare one answer better or the pair tied.Source: §4 · p. 6

4.1. Quantitative Results

  • StreamChat-7B produces equally or more preferable answers than VILA-1.5-40B in 77% of evaluation cases, despite using a much smaller LLM.StreamChat models also show superior streaming-interaction capabilities to LLaVA-Video and VILA; StreamChat-14B performs overall better than StreamChat-7B.Source: §4.1 · p. 6

4.2. Qualitative Results

  • In a qualitative example, StreamChat updates visual context as the stream continues, while VILA and LLaVA-Video show temporal misalignment and factual errors.The comparison concerns a question asked at a specific timestamp; previous methods use only visual context available up to that question.Source: §4.2 · p. 6

5. Benchmark Results

StreamChat is competitive on image and video benchmarks, including comparisons with models using larger base LLMs. Its cross-attention architecture supports efficient processing of more video frames, while omitting some high-resolution techniques may constrain performance on those benchmarks.

Scope

  • StreamChat omits multiple vision encoders and image tiling to maintain streaming efficiency, which could compromise performance on benchmarks requiring high-resolution inputs.Source: §5 · p. 7

Image benchmarks

  • StreamChat-7B scores 48.1 on MMMU, surpassing LLaVA-NeXT-8B and Cambrian-1-8B by 6.4 and 5.4 points, respectively.StreamChat also outperforms LLaVA-NeXT and Cambrian-1 on TextVQA while using significantly fewer visual tokens.Source: §5 · p. 7

Video benchmarks

  • Scores of 58.6/62.8 on VideoMME outperform VideoLLaMA2-7B by 10.7/11.5 points, and StreamChat-14B outperforms VILA-40B and VideoLLaMA2-72B.The supplied Table 2 caption identifies the video-benchmark comparison and notes that StreamChat’s cross-attention architecture is efficient when processing many frames.Source: §5 · p. 7, Table 2 · p. 7

Efficiency

  • The cross-attention architecture mitigates the heavy computation of self-attention across frames, keeping StreamChat efficient when processing more frames for inference.The supplied Table 2 caption also characterizes the architecture as efficient for processing a large number of video frames.Source: §5 · p. 7, Table 2 · p. 7

6. Ablation Studies

The ablations examine architectural choices and dense instruction data across image, video, and streaming evaluations. Results favor the final architecture and show that dense instruction data is particularly beneficial for streaming interaction.

Evaluation setup

  • The ablation study compares architectural designs and the dense instruction dataset across four image benchmarks, four video benchmarks, and the streaming evaluation.In the streaming evaluation, other models are reported relative to StreamChat, which is included as a candidate in the last row.Source: §6 · p. 7, Table 3 · p. 8

Architectural ablations

  • The final solution substantially outperforms the model without parameter reuse across all evaluated benchmarks, while not reusing LLM parameters causes significant training instability and poor final performance.The final solution also outperforms the version without V-FFN experts on 8 of 9 benchmarks; the linear gate improves performance on 6 of 9 benchmarks relative to the tanh gate, despite the tanh gate’s faster early convergence.Source: §6 · p. 8
  • A tanh gate facilitates faster early training convergence but yields poorer final performance than a linear gate.The linear gate improves performance on 6 of 9 benchmarks compared with the tanh gate.Source: §6 · p. 8

Dense instruction data

  • Training with dense instruction data yields equally or more preferable answers in 75% of streaming evaluation cases, while performance remains comparable on existing image and video benchmarks.The comparison is against the final solution trained without dense instruction data; the authors conclude that existing image or video instruction-tuning datasets alone are insufficient for effective streaming interactions.Source: §6 · p. 8

7. Related Works

Related work spans large multimodal models, offline video understanding, and streaming-video models. StreamChat’s distinction from the latter is dynamic visual-context updating during decoding.

Large Multimodal Models

  • Large multimodal models support tasks including image captioning [30] and visual question answering [16] [18] [40], with work spanning Flamingo [2], GPT-4V [1], and open-source instruction-tuned models.Subsequent examples include LLaVA-1.5 [31], Qwen-VL [5], and CogVLM [58].Source: §7 · p. 8
  • Recent open models including InternVL2 [10], LLaVA-OneVision [24], and Qwen2-VL [57] are reported to outperform closed models GPT-4o and Gemini-1.5-Pro [51].Source: §7 · p. 8

Multimodal Video Models

  • Many video LMM methods address long-video processing or spatial and temporal compression, but focus on offline understanding with the entire video available beforehand.StreamChat instead focuses on streaming-video processing.Source: §7 · p. 8

Streaming Video Models

  • Streaming-video approaches such as VideoLLM-online [7] and Flash-VStream [77] target fluency or responsiveness; StreamChat instead updates visual context during decoding.The passage contrasts this with methods that rely on fixed video content available up to the moment a question is posed.Source: §7 · p. 8

8. Conclusion

StreamChat combines per-step visual-context updates with a cross-attention architecture and V-FFN experts to capture streaming changes and produce temporally aligned responses. It also introduces training and temporal-positioning components, and performs competitively on established benchmarks while excelling in streaming interaction.

8. Conclusion

  • StreamChat uses a flexible cross-attention architecture with V-FFN experts and continuously updates visual context at each decoding step.The updates capture dynamic streaming content and lead to temporally aligned responses.Source: §8 · p. 9
  • A dense instruction dataset and parallel 3D-RoPE mechanism support training streaming interaction models and arranging streaming video and text.Source: §8 · p. 9
  • StreamChat achieves competitive performance on established image and video benchmarks and excels in streaming interaction scenarios.Source: §8 · p. 9

9. Limitations

StreamChat's text-token timestamps are generated heuristically from coarse-grained temporal annotations, which may reduce temporal alignment accuracy in complex videos requiring precise timing.

9. Limitations

  • Text-token timestamps are generated heuristically from coarse-grained temporal annotations rather than manually annotated.Source: §9 · p. 9
  • Heuristic timestamps may introduce temporal-alignment inaccuracies, particularly in complex video scenarios where precise timing is crucial.Source: §9 · p. 9
  • The stated limitation concerns timestamp precision and temporal alignment, not the broader streaming-interaction setting.Source: §9 · p. 9

Related papers

Loading 2412.08646v2…