Source-linked AI summary
OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction
Xiangyu Zeng, Yuandong Yang, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Qingyi Si, Dingyu Yao, Changlian Ma, Haoran Chen, Xinyu Chen, Yansong Shi, Junhao Zhou, Yifei Li, Jun Zhang, Chuanyu Qin, Chenxu Yang, Xinlei Yu, Kun Ouyang, Yuchen Shao, Qianshan Wei, Changhai Zhou, Jun Gao, Jiaqi Wang, Limin Wang
TL;DR
Streaming video LLMs must retain evidence before its relevance to future tasks is known while responding from evidence available so far. OneStreamer jointly learns evidence recording and task response through proactive generation, with PHCM memory and PSTL state supervision. Its 4B model achieves the best results across eight evaluated benchmarks, and ablations support the value of retained captions and selective state supervision.
Problem
Streaming video models must form reusable factual memory from observations of unknown future relevance while preserving real-time perception and responding when evidence warrants it.
Source: §1 · p. 1Method
OneStreamer uses shared proactive generation for task response and query-independent recording, with PHCM caption memory, PSTL state supervision, and evidence-timed streaming data.
Source: §1 · p. 2Results
The 4B model achieves the best results across all eight evaluated benchmarks; retaining captions improves historical QA, and PSTL outperforms dense state supervision while supervising 27.5% of annotated state tokens.
Source: §1 · pp. 2–3Takeaways & Limitations
The results support proactive generation as a shared mechanism connecting visual perception, reusable factual memory, and timely task response.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Streaming video LLMs must retain evidence before its relevance to future tasks is known and respond when sufficient evidence becomes available. The challenge is to form reusable factual memory without compromising real-time perception. We introduce OneStreamer, which jointly learns query-independent evidence recording and task response through a shared proactive generation process. Its Proactive Hierarchical Caption Memory (PHCM) produces time-grounded local-detail captions and summaries of completed events. Streaming caption targets supervise the interpretation of observed video prefixes during training. At inference, model-generated records complement a recent visual window, providing reusable factual context without revisiting historical visual features. Proactive State Transition Learning (PSTL) reduces the dominance of repeated waiting states by preserving supervision at all output anchors and selecting representative state-change and state-persistence tokens. We further develop a streaming data synthesis pipeline that aligns output content and timing with available evidence. Combining the resulting streaming captions and QA with cleaned open-source data yields OneStreamer-1M, a broad-coverage streaming video interaction dataset with over one million records spanning diverse tasks. Our 4B model achieves the best results among the compared methods across all eight evaluated streaming video understanding benchmarks. Ablations show that retaining generated captions improves historical QA without degrading real-time perception. PSTL also outperforms dense state supervision while supervising only 27.5% of annotated state tokens. Together, these results support proactive generation as a shared learning interface connecting perception, memory formation, and timely response in streaming video interaction.
1 Introduction
Streaming video LLMs must preserve evidence that may matter to future tasks while grounding timely responses in what has been observed. OneStreamer unifies query-independent evidence recording and task response through proactive generation, and its 4B model achieves the best results across eight evaluated benchmarks.
1 Introduction
- Streaming models must retain observations whose future relevance is unknown while grounding responses in evidence available so far.Historical visual tokens can compete with recent observations for limited context, while reasoning traces do not necessarily form reusable factual records and dense state supervision can overemphasize repeated waiting states.Source: §1 · pp. 1–2
- The 4B OneStreamer model achieves the best results across all eight evaluated benchmarks, with average relative improvements of 25.0% over Qwen3-VL and 6.1% over the strongest competing method on each benchmark.The benchmarks cover streaming video understanding and proactive response.Source: Figure 1 · p. 2, §1 · pp. 2–3
- OneStreamer jointly learns when to record or respond and what to generate through task-specific control tokens and associated text in one causal visual-language sequence.Proactive generation is used for both user-visible responses and reusable factual records.Source: §1 · p. 2
- PHCM combines time-aligned local captions and completed-event summaries with a recent visual window, retaining factual context without revisiting historical visual features.Streaming caption targets supervise interpretation of observed video prefixes during training.Source: §1 · p. 2
- The data pipeline aligns caption and QA content and timing with available evidence, combining the resulting examples with cleaned open-source data in OneStreamer-1M.The dataset is a broad-coverage training corpus for streaming video interaction.Source: §1 · p. 2
2 Related Work
Related work addresses online video memory and streaming response through distinct approaches. OneStreamer combines query-independent factual recording with task response.
2.1 Perception and Memory in Video Understanding
- Video memory approaches extend accessible history through compression, context extension, token selection, retrieval, structured memory, or recent windows, but face a trade-off between retaining history and preserving current visual detail.OneStreamer complements recent visual inputs with proactively generated, time-grounded factual records for later task-conditioned use.Source: §2.1 · p. 3
2.2 Streaming Description and Proactive Response
- Streaming description, reasoning, and proactive-response methods address online generation or response timing, but reasoning traces are not necessarily reusable factual records and relevant evidence need not warrant an immediate response.OneStreamer jointly models factual recording and task response with selective state supervision.Source: §2.2 · p. 3
3 Method
OneStreamer processes streaming video through a shared causal generation process that determines when to record evidence or respond and what text to generate. PHCM supplies reusable caption memory, while PSTL selectively supervises state tokens.
3 Method
- The shared causal visual-language process predicts task-specific control tokens and generates associated text when a predicted state initiates an output.It processes ordered clips using recent visual tokens interleaved with accumulated captions and dialogue.Source: §3 · p. 3, §3.1 · p. 3
3.1 Overall Architecture
- Recent visual tokens are interleaved with local captions, event summaries, and dialogue; control tokens distinguish recording, continued observation, and user-visible responses.For proactive QA, Standby marks emerging but insufficient evidence, while Response initiates an answer.Source: Figure 3 · p. 4, §3.1 · p. 4
3.2 Proactive Hierarchical Caption Memory
- PHCM complements a recent visual window with time-aligned local-detail captions and sparser summaries of completed events, preserving distant context without retrieving historical visual features.Both record types describe only video content observed so far and are retained for subsequent predictions.Source: §3.2 · p. 4
- During training, causal streaming caption targets supervise interpretation of observed prefixes; at inference, retained records become explicit context for later predictions.This makes streaming description a source of query-independent factual memory.Source: §3.2 · p. 4
3.3 Proactive State Transition Learning
- PSTL preserves supervision at output anchors and selects representative state-change and state-persistence tokens, while unselected tokens remain in the sequence but are excluded from state loss.Its quota balances transition groups within a sequence, and masking leaves caption and task-text supervision unchanged.Source: §3.3 · p. 5
4 Dataset Construction
OneStreamer-1M is built by combining synthesized streaming captions and QA with cleaned open-source data. The pipeline aligns outputs with evidence available at their assigned times.
4 Dataset Construction
- OneStreamer-1M combines synthesized streaming examples with cleaned open-source data to form a broad-coverage dataset for streaming video interaction.Its construction includes complementary caption and QA synthesis branches.Source: §4 · p. 5
4.1 Streaming Caption Synthesis
- Caption synthesis releases local descriptions only after supporting intervals are observed and segment summaries only after their events or segments are complete.The pipeline curates videos, verifies multi-granularity captions against source video, and converts them into evidence-aligned streaming targets.Source: §4.1 · pp. 5–6
4.2 Streaming QA Synthesis
- QA synthesis calibrates each response to the earliest timestamp at which available evidence sufficiently supports the answer.Examples are grounded in localized evidence intervals; the resulting sequence uses silence before relevant evidence, standby as evidence emerges, and response at the calibrated time.Source: §4.2 · pp. 6–7
5 Experiments
OneStreamer achieves the highest aggregate scores among compared methods across the eight evaluated benchmarks. Its ablations find that caption memory supports historical QA and perception, while PSTL improves proactive response with sparse supervision.
5 Experiments
- The 4B model is initialized from Qwen3-VL-4B-Instruct and fine-tuned for one epoch on OneStreamer-1M plus offline LLaVA-Video data.The vision encoder is frozen; training uses 32 NVIDIA H200 GPUs and a maximum sequence length of 131,072 tokens.Source: §5 · p. 7
5.1 Main Results
- OneStreamer achieves the highest aggregate scores among compared methods on four perception and memory benchmarks and four proactive-response benchmarks.Scores include 72.1 on OVOBench, 86.9 on StreamingBench Real-Time, 66.8 on OVBench, and 71.3 on ODVBench; proactive-response scores are 48.7 on ProactiveVQA, 36.6 on OmniMMI with ASR, 41.6 on OVO-Timing, and 2.87 on ViSpeak.Source: §5.1 · p. 7
5.2 Ablations and Findings
- On OVOBench-Backward, retaining PHCM records raises ASI from 63.5 to 71.6 and EPM from 62.0 to 62.6 after source frames leave the Recent-16 window.The compared settings use the same checkpoint and differ only in whether generated caption records are retained.Source: §5.2 · p. 8
- With Recent-16, PHCM improves OVOBench Real-Time from 80.9 to 81.4 and StreamingBench Real-Time from 86.3 to 86.9, while exceeding Full on ASI by 4.0 points.PHCM scores 71.6 ASI versus Full's 67.6, but its EPM is slightly lower: 62.6 versus 63.0.Source: §5.2 · p. 8, Table 2 · p. 8
- PSTL achieves 48.7 on ProactiveVQA, 36.6 on OmniMMI, and 41.6 on OVO-Timing while supervising 27.5% of annotated state tokens, outperforming dense supervision and ratio-matched Random Sparse.It preserves supervision at all output anchors and selects representative state-change and state-persistence tokens; Transition Only trails it on OmniMMI and OVO-Timing.Source: §5.2 · p. 8, Table 3 · p. 8
5.3 Efficiency and Latency
- On a 360 s OVOBench sample, PHCM reduces Full's context length by 93.1%, GPU memory by 60.4%, and TTFT from 4.560 s to 0.124 s.Compared with FIFO, PHCM adds 1,272 context tokens, 0.29 GB of GPU memory, and 0.030 s of TTFT.Source: §5.3 · p. 9, Table 4 · p. 9
5.4 Qualitative Analysis
- In the qualitative examples, PHCM retains details needed to answer after their frames leave the visual window, while response states update a count as evidence arrives.The examples show a later answer about where children can read and proactive counting from one to three.Source: §5.4 · p. 9, Figure 6 · p. 9
6 Conclusion
OneStreamer unifies query-independent evidence recording and task response through proactive generation. The 4B model leads the compared methods across eight benchmarks, and ablations support caption memory and selective state supervision as parts of this shared learning approach.
6 Conclusion
- OneStreamer jointly learns query-independent evidence recording and task response through proactive generation.The 4B model achieves the best results among compared methods across all eight benchmarks.Source: §6 · p. 10
- PHCM adds time-grounded local-detail captions and completed-event summaries to a recent visual window, while PSTL selects representative state-change and persistence tokens.The streaming data pipeline aligns caption and QA targets with available evidence to construct OneStreamer-1M.Source: §6 · p. 10
- Ablations show that generated captions improve historical QA without degrading real-time perception, and PSTL beats dense supervision and a supervision-matched random baseline.The findings support proactive generation as a shared mechanism for reusable factual memory and timely responses grounded in past and current evidence.Source: §6 · p. 10
A More Details of Method
PSTL selects state-token supervision for state changes and persistence while retaining supervision at output anchors. At inference, PHCM appends generated captions to a recent visual window, allowing records to remain available after their source frames leave that window.
A.1 Proactive State Transition Learning
- PSTL applies to task-specific control-state sequences, with output anchors including Observe and Summary for PHCM and Response for proactive QA and interaction.Table 5 summarizes the task-specific control-state vocabulary and anchor sets.Source: §A.1 · p. 14, Table 5 · p. 14
- For each source state, PSTL retains supervision for changes and persistence; Silence may persist until sufficient evidence supports an output-state transition.Consecutive Observe tokens can initiate distinct local-detail captions, and the shared quota also caps groups without a direct transition to an output anchor.Source: §A.1 · p. 14
- Unselected control tokens remain in the teacher-forced sequence and can condition later predictions, but do not contribute to state loss at their own indices.Caption text and task outputs retain full next-token supervision, and PSTL is not applied at inference.Source: §A.1 · p. 14
A.2 Online Inference with PHCM
- At each inference step, OneStreamer combines current visual tokens with retained dialogue and generated caption records, appending new records when it predicts Observe or Summary.Records remain available after source frames leave the FIFO window; proactive QA uses the same history with the user instruction.Source: §A.2 · p. 14
B More Details of Dataset
OneStreamer-1M combines diverse streaming-interaction supervision with caption and QA sequences designed to release information only when supporting evidence is available. Its caption records preserve local details and completed events, while its QA examples receive evidence-grounded response times.
B.1 Composition and Statistics
- The dataset combines four supervision families, with 630,850 Perception & Memory QA records, 271,570 Proactive Interaction records, 197,584 Proactive QA records, and 60,000 Proactive Caption Memory records.Together, these families supervise evidence recording, online perception and memory, and proactive task response.Source: §B.1 · pp. 14–16
B.2 Streaming Caption Construction
- Caption construction verifies multi-granularity annotations, then releases local-detail </Observe> targets after their supporting intervals and </Summary> targets when events are complete.Video-level captions instead supervise full-video summarization after the stream ends; representative examples span sports, cooking, and gameplay.Source: §B.2 · p. 16, Figure 7 · p. 17, Figure 8 · p. 18
B.3 Streaming QA Construction
- Streaming QA construction generates task-directed question–answer pairs, verifies that localized evidence supports each answer, and calibrates response times against preceding visual windows.The calibration selects an early near-peak candidate and validates answerability using the video prefix up to that time.Source: §B.3 · p. 20
C.1 Training
The paper specifies supervised fine-tuning configuration and an inference procedure that supplies only available video context and task-permitted history. Evaluation uses benchmark-specific sampling and configurable generation settings.
C.1 Training
- Supervised fine-tuning uses the instruction-tuned checkpoint with registered response-control and summary tokens, and runs on 32 NVIDIA H200 GPUs across four nodes.The setup uses PyTorch FSDP, FlashAttention-2, and full activation recomputation; configured epochs and optimizer steps define the schedule.Source: §C.1 · p. 21, Table 7 · p. 21
C.2 Evaluation
- Inference samples videos at 1–4 FPS, dynamically resizes frames, and supplies only the available clip or stream prefix with the task prompt and permitted dialogue history.Inputs preserve aspect ratio and satisfy processor spatial-alignment requirements under configured frame and pixel budgets.Source: §C.2 · p. 21
- Greedy decoding is the default, each generation call has its own max_new_tokens limit, and independent samples are distributed across eight GPUs before outputs are merged.Prediction generation and metric computation are separate stages, with local scorers operating on parsed answers or response timestamps.Source: §C.2 · p. 21
D Per-benchmark Results
The evaluation section describes comparison-model protocols and benchmark-specific inference settings, then tests caption supervision, hierarchical memory, and online memory-construction efficiency. The ablations report gains from trained caption memory and show that it is substantially less costly than retaining full visual history.
D Per-benchmark Results
- Comparison models with reported results use their published metrics; re-evaluated models are tested under both original and OneStreamer settings, with the better benchmark-level aggregate reported.OneStreamer uses benchmark-specific frame sampling and context configurations for perception and proactive-response evaluations.Source: D Per-benchmark Results · p. 22
E.1 Effect of Proactive Caption Supervision
- Caption supervision improves aggregate scores across all six visual-only evaluation settings, including full-history and recent-window QA, even when caption memory is not used at inference.The comparison removes 60K proactive caption examples from the ablated model’s training.Source: §E.1 · p. 25, Table 16 · p. 25
- 2.0 percentage points on OVOBench Overall and 0.6 points on StreamingBench Real-Time are the gains from adding PHCM to OneStreamer’s recent visual window.Retaining descriptions from the caption-ablated model changes scores by −0.1 and +0.3 points relative to FIFO, respectively.Source: §E.1 · p. 25, Table 17 · p. 25
E.2 Effectiveness of Hierarchical Caption Memory
- 72.1 on OVOBench Overall and 86.9 on StreamingBench Real-Time are achieved by combining </Observe> and </Summary> records, the highest OVOBench score and tied-best StreamingBench score.The two record types individually improve both metrics over using no Caption Memory; event-level summaries add complementary information to local-detail records.Source: §E.2 · pp. 25–26, Table 18 · p. 26
E.3 Efficiency of Online Memory Construction
- 84.2% lower mean update latency, 65.3% lower peak GPU memory, and 90.4% lower peak input length than Full; PHCM averages 0.636 s per update.The throughput test replays a 360 s clip at one update per second using a 128-frame visual context for PHCM and FIFO.Source: §E.3 · p. 26, Table 19 · p. 26
E.4 Effectiveness of Training Data Construction
Progressive training-data additions improve aggregate performance, with the complete mixture achieving the strongest five-benchmark average. Separate examples illustrate how query-independent caption records can supply earlier visual evidence for later questions.
E.4 Effectiveness of Training Data Construction
- The progressive mixture adds TRA-QA, then FPPR, CQTC refinements, and finally SPM, which supplies additional general, spatial, answerability, procedural, and counting QA.These components contribute complementary training data for understanding and proactive response.Source: §E.4 · pp. 26–27
- 56.0 is the highest five-benchmark average, achieved by the complete OneStreamer training mixture after SPM raises OVOBench from 71.1 to 72.1 and StreamingBench Real-Time from 85.3 to 86.9.The best individual proactive-response scores occur at intermediate stages, while the full mixture has the strongest aggregate performance.Source: §E.4 · p. 27, Table 20 · p. 27
- Without adding samples, CQTC raises OVO-Timing from 33.0 to 42.0 and the five-benchmark average from 53.4 to 55.6 through content verification and response-time calibration.At this stage, ProactiveVQA rises from 47.3 to 49.0 and OmniMMI from 29.9 to 30.8.Source: §E.4 · p. 27
F.1 Memory and Perception
- Caption memory records object locations, attributes, and scene context before questions arrive, enabling later recall of details such as a wreath’s location or a light switch’s color.The examples show query-independent recording followed by task-conditioned use.Source: §F.1 · p. 27
F.2 Proactive Response
Proactive responses track when task-relevant evidence becomes sufficient, while the failure example shows that an initial action misinterpretation may go uncorrected in the shown sequence.
F.2 Proactive Response
- At 00:52, the model responds when the walking scene and requested handle appear together; in another example, it answers at 00:26 after waiting while dough is shaped into a ring.The examples distinguish emerging evidence from evidence sufficient to support the requested output.Source: §F.2 · pp. 27–28, Figure 11 · p. 29
- Figure 10 shows time-aligned observation and summary records being retained to answer questions about earlier observations.Source: Figure 10 · p. 28
- In repeated-output tasks, OneStreamer describes cooking actions incrementally and updates a tearing-event count from one to four, with silence between updates.The count updates occur at 00:04, 00:08, 00:10, and 00:15.Source: §F.2 · p. 28
F.3 Failure Cases
- At 00:10, the model incorrectly says a person is closing the printer, then remains silent through 00:13 without correcting the description despite later frames showing it open.The case highlights challenges in interpreting object-state changes and correcting earlier responses.Source: §F.3 · pp. 28–29, Figure 12 · p. 29