PaperlayerView exact revision on arXiv

Source-linked AI summary

Streaming Video Instruction Tuning

Jiaer Xia, Peixian Chen, Mengdan Zhang, Xing Sun, Kaiyang Zhou

arXiv:2512.21334v2cs.CV

TL;DR

Real-time assistants must handle continuous, unbounded video streams while responding to dynamic instructions under latency constraints. Streamo combines end-to-end response timing with a multi-task instruction dataset, and experiments show strong performance across streaming and offline benchmarks, including a +13.83% average gain over Dispider. The paper positions this combination as a step toward general-purpose interactive video assistants.

  • Problem

    Existing offline video models process complete clips, whereas real-time assistants must interpret unbounded streams, manage response timing, and support dynamic instructions.

  • Method

    Streamo combines end-to-end response-state prediction with Streamo-Instruct-465K, a temporally annotated multi-task dataset covering narration, captioning, grounding, and time-sensitive question answering.

  • Results

    Streamo outperforms existing online approaches across streaming and offline benchmarks, while Streamo-7B exceeds Dispider by +13.83% on average performance.

  • Takeaways & Limitations

    Streamo provides a unified real-time assistant framework with temporal awareness, frame-level decision-making, and multi-task instruction-following.

  • Takeaways & Limitations

    The pipeline lacks specialized long-sequence optimizations, causing memory and latency costs that become prohibitive as sequence length grows.

Abstract

from arXiv · show

We present Streamo, a real-time streaming video LLM that serves as a general-purpose interactive assistant. Unlike existing online video models that focus narrowly on question answering or captioning, Streamo performs a broad spectrum of streaming video tasks, including real-time narration, action understanding, event captioning, temporal event grounding, and time-sensitive question answering. To develop such versatility, we construct Streamo-Instruct-465K, a large-scale instruction-following dataset tailored for streaming video understanding. The dataset covers diverse temporal contexts and multi-task supervision, enabling unified training across heterogeneous streaming tasks. After training end-to-end on the instruction-following dataset through a streamlined pipeline, Streamo exhibits strong temporal reasoning, responsive interaction, and broad generalization across a variety of streaming benchmarks. Extensive experiments show that Streamo bridges the gap between offline video perception models and real-time multimodal assistants, making a step toward unified, intelligent video understanding in continuous video streams.

1. Introduction

The paper targets real-time video assistants that must interpret unbounded streams, manage response timing, and support diverse tasks. Streamo addresses these challenges with end-to-end decision-making, a multi-task temporally annotated dataset, and strong streaming-benchmark performance.

  • 1. Introduction: Streaming video assistants must continuously process unbounded data, detect task-relevant moments, and choose response timing and granularity as events unfold.These requirements differ from offline models that process complete clips before producing an output.
  • 1. Introduction: Streamo unifies frame-level response-state prediction and response generation end-to-end, using Silence, Standby, and Response heads for fine-grained output timing.The triggered response is generated immediately in one pass.
  • 1. Introduction: Streamo-Instruct-465K standardizes temporal annotations and response granularity across narration, captioning, grounding, and time-sensitive question answering.The dataset is designed for streaming video understanding and interaction.
  • 1. Introduction: The paper introduces Streamo-Bench to evaluate instruction understanding across diverse interactive streaming tasks.The benchmark supports research on streaming video instruction-following.

2. Related Work

Existing video LLMs have advanced offline comprehension, including long-video reasoning, but require complete videos before producing outputs. This offline design limits their ability to respond continuously within streaming video.

  • 2. Related Work: Recent video LLMs can analyze complete, pre-recorded videos and reason over long temporal sequences for captioning, summarization, and question answering.Examples include models that process videos spanning several hours and perform complex video reasoning.
  • 2. Related Work: Offline video LLMs require the entire video as input before output, preventing continuous-stream handling and precise timing of responses.Their single-pass design lacks mechanisms for identifying when to generate during an ongoing stream.

3. Streamo: Architecture and Training

Streamo reformulates offline video understanding as streaming, multi-turn interaction with frame-level response states. Its training uses state-aware weighting to address severe imbalance in response timing labels.

  • 3.2. Data Structure: Streaming video is processed from partial observations, requiring models to handle continuous data and variable response timing without future frames.
  • 3.2. Data Structure: Videos are segmented into contiguous temporal turns, with questions and responses inserted at task-appropriate points in a multi-turn dialogue.
  • 3.3. Training: Three response states—Silence, Standby, and Response—are integrated into next-token prediction to support frame-level decisions and parallel supervised training.
  • 3.3. Training: More than 80% of typical state tokens are Silence, biasing the model toward nonresponse and making response timing difficult to learn.
  • 3.3. Training: Focal weighting and frequency-based alpha weights jointly emphasize difficult and infrequent state tokens in the cross-entropy loss.

4. Streamo-Instruct-465K

Streamo-Instruct-465K standardizes temporal, multi-task supervision for streaming video understanding. It combines curated annotations and existing datasets across diverse tasks and video durations.

  • 4.1. Data Construction: Streamo-Instruct-465K applies a unified annotation protocol and assigns multiple task annotations to videos for clear response-decision supervision.
  • 4.1. Data Construction: The dataset distribution summarizes task composition and video-duration coverage across the constructed corpus.
  • 4.1. Data Construction: The dataset covers real-time narration, event captioning, action captioning, event grounding, and time-sensitive question answering.
  • 4.1. Data Construction: 400K valid curated samples were combined with offline video QA data to form Streamo-Instruct-465K from 135,875 videos sourced from multiple open datasets.

5. Experiments

Streamo is evaluated across online, offline, and multi-instruction streaming benchmarks using unified training settings. The results show strong online performance, dataset advantages, retained offline capability, and sensitivity to instruction complexity and class imbalance.

  • Comparison with SOTA on Online Video Benchmarks: +13.83% average performance over Dispider establishes Streamo-7B as the strongest reported model on OVO-Bench.The model trained at 1 fps also gains +4.66% when evaluated at 2 fps without retraining.
  • Comparison with SOTA on Online Video Benchmarks: +7.1% on the forward task and +11.79% overall show Streamo-Instruct-465K outperforming ET-Instruct-164K.Holding architecture and training setup constant, the dataset also outperforms ET-Instruction and LLaVA-Video by +7.8% and +2.5% on average, respectively.
  • Comparison with SOTA on Online Video Benchmarks: Adding offline LLaVA-Video supervision improves real-time perceptual accuracy but compromises streaming ability, revealing a trade-off in offline-only supervision.The results also indicate that Streamo-Instruct-465K transfers effectively to online scenarios while retaining strong offline perceptual capability.
  • Streamo-Bench: Streamo-Bench evaluates forward and backward grounding, narration and dense captions, and time-sensitive question answering within multi-instruction videos.Existing online models show limited comprehensive task coverage, with failures linked to complex instructions, open-ended prompts, and changing-answer questions.
  • Ablation: Focal loss dynamically reweights token losses using hardness and per-batch state frequency, improving performance over vanilla cross-entropy and fixed-weight baselines.The state-label ratio is approximately Silence:Standby:Response = 12:3:2, creating imbalance that biases conventional training toward Silence.

6. Conclusion

The paper jointly advances streaming-video model training and data construction through Streamo, Streamo-Instruct-465K, and Streamo-Bench. The resulting system performs strongly on streaming tasks while retaining competitive offline perception and multitasking capability.

  • 6. Conclusion: Streamo combines an end-to-end training framework, Streamo-Instruct-465K, and Streamo-Bench to support general-purpose real-time video assistants.The framework converts multiple state-of-the-art offline models into online versions and evaluates robust multitasking.

7. Limitations and Future Work

Streamo’s accuracy is constrained by unbounded temporal context. Without specialized long-sequence optimization, memory and latency costs become prohibitive as sequences grow.

  • 7. Limitations and Future Work: Unbounded temporal context creates significant memory and latency costs because the current pipeline lacks specialized long-sequence optimizations.Future work proposes KV-cache management, visual-token pruning, sliding-window attention, and adaptive frame compression.

Supplementary Material

Streamo uses dedicated prompting and diverse task templates to adapt offline models to dynamic streaming video. These prompts encode response states and support broader multi-task instruction following beyond standalone question answering.

  • Supplementary Material: A dedicated system prompt teaches Streamo to interpret three response states and make frame-level real-time decisions.The prompt is designed to help the model adapt to streaming input patterns and perform the required behavior transformation.
  • Supplementary Material: Diverse task templates broaden instruction following across streaming tasks beyond prior standalone QA setups.The templates are intended to improve understanding of task requirements and general multi-task capabilities.

A.3. More Experimental Results

Streamo extends offline video models to real-time streaming through minimally intrusive training, while achieving strong response timing and preserving offline capabilities across base models. Additional experiments analyze its benchmark performance, three-state design, and multi-task evaluation protocol.

  • Additional Base-Model Results: Streamo’s framework consistently enables strong real-time streaming performance across InternVL3 and Qwen3VL while preserving their underlying offline capabilities.The supplementary evaluations report online and offline benchmark results for both base models.
  • ViSpeak-Bench: Streamo achieves a clear advantage in response-time accuracy on ViSpeak-Bench.
  • Three-State Ablation: The [Standby] state marks relevant frames before the outcome is ready, preserving attention across the event interval and improving temporal coverage for grounding.The model switches from [Silence] to [Standby] when the relevant event begins, then emits [Response] after the outcome becomes clear.
  • Three-State Ablation: Replacing the three-state design with EOS-only training causes notable drops, especially on proactive FAR and grounding tasks.The comparison uses the same Streamo-Instruct training dataset and changes only the response-state design.
  • Streamo-Bench: Streamo-Bench combines Grounding, Narration, Caption, and Time-Sensitive QA across 300 videos and 3,000 task-specific instances.Grounding includes forward and backward tasks, while the remaining instances include dense captioning, narration, and TSQA.
  • Evaluation Metrics: Streamo-Bench evaluates grounding with mIoU, narration and captioning with pairwise win rate against Qwen2.5-VL-72B, and TSQA using content and timestamp correctness.Grounding compares predicted and ground-truth temporal intervals, while TSQA requires both answer content and timing to be correct.

C.4. Further Analysis

Further analysis shows that existing models often fail to follow diverse streaming task instructions, while Streamo uses explicit response states and multi-task outputs to support task-appropriate interaction. The accompanying figures and tables document its state transitions, benchmark design, prompts, and outputs.

  • Instruction-Following Analysis: Existing models frequently fail to distinguish streaming task types, falling back to generic descriptions for grounding and mishandling real-time updates in TSQA.Models trained mainly on captioning or QA often produce task-specific outputs only for those familiar tasks.
  • Instruction-Following Analysis: Streamo-Bench is designed to test instruction-following across multiple open-ended streaming tasks rather than QA alone.The benchmark specifically targets task-specific interpretation and response in streaming scenarios.
  • Three-State Decoding: The three-state decoder stays in [Silence] for irrelevant frames, enters [Standby] when a relevant event begins, and emits [Response] after the outcome is clear.For grounding, persistent [Standby] preserves attention over the relevant interval and supports fuller event-span coverage.
  • Benchmark and Prompt Design: The supplementary materials provide task prompts for event-caption rewriting, diverse streaming tasks, frame-level descriptions, narration merging, and Time-Sensitive QA.
  • Output Visualization: Visualizations show Streamo producing suitable outputs for unseen instructions with different response granularities across TSQA, narration, and caption tasks.Arrows identify response moments, and colors distinguish task types in the visualized outputs.
Loading 2512.21334v2…