Source-linked AI summary
TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos
Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, Lingpeng Kong, Qi Liu, Yuanxing Zhang, Xu Sun
TL;DR
Streaming VideoLLMs must process long, highly redundant frame streams while answering queries in real time. TimeChat-Online uses Differential Token Drop and a streaming-specific dataset, achieving state-of-the-art streaming performance while eliminating up to 82.8% of visual tokens.
Problem
Streaming VideoLLMs must continuously process long, highly redundant frame streams while supporting immediate interaction, including backward tracing, current perception, and future responding.
Method
TimeChat-Online combines Differential Token Drop, which filters temporally redundant visual tokens, with the TimeChat-Online-139K instruction-tuning dataset for streaming interactions.
Results
TimeChat-Online achieves state-of-the-art streaming performance and superior offline long-video understanding while eliminating up to 82.8% of visual tokens.
Takeaways & Limitations
The results reveal substantial redundancy in streaming videos and support computationally efficient video encoding for long-form and real-time understanding.
Abstract
from arXiv · showhide
The rapid growth of online video platforms, particularly live streaming services, has created an urgent need for real-time video understanding systems. These systems must process continuous video streams and respond to user queries instantaneously, presenting unique challenges for current Video Large Language Models (VideoLLMs). While existing VideoLLMs excel at processing complete videos, they face significant limitations in streaming scenarios due to their inability to handle dense, redundant frames efficiently. We introduce TimeChat-Online, a novel online VideoLLM that revolutionizes real-time video interaction. At its core lies our innovative Differential Token Drop (DTD) module, which addresses the fundamental challenge of visual redundancy in streaming videos. Drawing inspiration from human visual perception's Change Blindness phenomenon, DTD preserves meaningful temporal changes while filtering out static, redundant content between frames. Remarkably, our experiments demonstrate that DTD achieves an 82.8% reduction in video tokens while maintaining 98% performance on StreamingBench, revealing that over 80% of visual content in streaming videos is naturally redundant without requiring language guidance. To enable seamless real-time interaction, we present TimeChat-Online-139K, a comprehensive streaming video dataset featuring diverse interaction patterns including backward-tracing, current-perception, and future-responding scenarios. TimeChat-Online's unique Proactive Response capability, naturally achieved through continuous monitoring of video scene transitions via DTD, sets it apart from conventional approaches. Our extensive evaluation demonstrates TimeChat-Online's superior performance on streaming benchmarks (StreamingBench and OvOBench) and maintaining competitive results on long-form video tasks such as Video-MME and MLVU.
1 Introduction
TimeChat-Online targets streaming video understanding by addressing redundant long-form context and the need for real-time, proactive interaction. Its Differential Token Drop mechanism selectively retains salient temporal changes, reducing visual tokens while preserving StreamingBench performance.
- Challenges: Streaming video understanding must handle high-rate, potentially infinite streams with substantial redundancy while supporting real-time interaction and proactive responses.Neighboring frames often repeat backgrounds and static objects, creating long-form context that must be maintained continuously.
- Limitations of Existing Methods: Existing VideoLLMs are primarily designed for offline processing and struggle with proactive responses and long-form, high-redundancy streaming videos.Their limitations include sparse frame sampling in short-context models and difficulty processing continuous streams.
- Contributions: TimeChat-Online is introduced as an online VideoLLM for efficient real-time interaction with streaming video content.The method is designed to address both long-form redundancy and continuous interaction challenges.
- Contributions: Differential Token Drop selectively preserves changed visual tokens between successive frames while filtering static, redundant content.The mechanism is inspired by change blindness in human visual perception and operates on spatial-temporal changes.
- Contributions: 82.8% of streaming-video tokens are eliminated with a 1.76× speed-up while maintaining 98% accuracy on StreamingBench.The paper attributes this reduction and efficiency gain to the proposed Differential Token Drop module.
2 Related Work
Prior streaming-video methods primarily improve efficiency through memory banks, KV-cache optimization, or token-count reduction, often using language guidance to select relevant content. TimeChat-Online instead introduces purely visual Differential Token Drop (DTD), while broader video-token pruning methods include fixed-length compression and adaptive token merging.
- Streaming Video Understanding: Streaming-video understanding has developed along two directions: efficient encoding of dense streams and enhanced real-time interaction.The cited passage identifies these as the two primary directions following VideoLLM-online.
- Streaming Video Understanding: Memory-bank methods retain informative video tokens, while ReKV and Inf-MLLM optimize KV-cache management and VideoLLM-MoD reduces token counts with Mixture-of-Depth.Examples include VideoStreaming, Flash-VStream, StreamChat, VideoChat-online, ReKV, Inf-MLLM, and VideoLLM-MoD.
- Streaming Video Understanding: Most existing streaming methods use language guidance from user queries to select relevant video content, whereas DTD reduces video tokens by over 80% using purely visual processing before language-model inference.DTD is presented as an early step that lightens the computational burden of video tokens.
- Efficient Video Token Pruning: Long-video token-pruning methods include fixed-size frame or clip compression that disregards video-specific dynamic redundancy, such as Llama-vid’s two-token representation per frame.The passage contrasts fixed compression with approaches designed to address dynamic visual redundancy.
- Efficient Video Token Pruning: Other pruning methods use adaptive token merging in spatial or temporal dimensions to better accommodate variation in video content.The supplied passage introduces adaptive token-merging approaches but truncates their further description.
3 TimeChat-Online Framework
TimeChat-Online is an online VideoLLM for timestamped Streaming VideoQA that continuously processes incoming frames, preserves temporal changes with Differential Token Drop, and supports proactive responses. Its framework combines position-aware redundancy removal with the TimeChat-Online-139K dataset covering backward tracing, real-time perception, and forward active responding.
- 3.1 Streaming VideoQA: TimeChat-Online continuously updates incoming video frames and uses current and past streams to answer timestamped Streaming VideoQA queries.For future-oriented questions, it can proactively respond when historical content is insufficient.
- 3.2 Differential Token Drop: DTD reduces streaming redundancy by preserving significant temporal changes through patch encoding, static redundancy calculation, and position-aware token dropping.It compares temporally consecutive, spatially aligned patches or visual tokens using pixel- and feature-level similarity.
- 3.2 Differential Token Drop: DTD preserves remaining tokens’ spatial-temporal relations with M-ROPE after dropping redundant current-frame tokens, enabling position-aware processing across the timeline.The remaining tokens retain their original 3D temporal, height, and width positions despite token removal.
- 3.3 TimeChat-Online-139K: TimeChat-Online-139K trains online VideoLLMs on diverse interactions spanning backward tracing, real-time perception, and forward active responding.The dataset is designed to address limited question-answer diversity in prior transformed streaming dialogue datasets.
- 3.3 TimeChat-Online-139K: Proactive responses are triggered at scene transitions, which DTD monitors through valleys in the video token drop-ratio curve.The model uses these trigger times to decide when to respond to future-responding questions.
4 Experiments
TimeChat-Online achieves strong streaming and offline video-understanding performance while substantially reducing visual tokens through DTD. Experiments further identify video-aware feature-level dropping as the most effective pruning design and demonstrate proactive future responding.
- Streaming VideoQA: 75.28 on StreamingBench establishes state-of-the-art real-time visual understanding performance, improving by 7.65 points.The result is reported for the Real-time Visual Understanding subtask.
- Streaming VideoQA: 75.28 vs. 73.68 with 44.2% fewer video tokens, while 82.8% dropping still yields 73.64 vs. 73.68 for full-token Qwen2.5VL-7B.These comparisons show substantial redundancy in 1 FPS streams while preserving performance under aggressive dropping.
- Streaming VideoQA: 45.6 remains achievable after dropping 84.8% of video tokens, and DTD proactively responds when future scene transitions trigger relevant visual changes.Figure 4 illustrates responses generated at future trigger times marked by low token-drop ratios.
- Offline Long-Form Video Understanding: TimeChat-Online outperforms recent online VideoLLMs on VideoMME, MLVU, and LongVideoBench in offline long-form video understanding.The comparison includes VideoChat-Online and Dispider-7B, with DTD providing the efficiency basis.
- Ablations and Training Strategy: Feature-level, video-aware dropping—the final DTD design—achieves the best zero-shot performance at the same dropping ratio, showing that joint spatio-temporal pruning is most effective.Table 4 compares VisionZip, pixel-level dropping, feature-level dropping, and frame-aware versus video-aware selection.
5 Analysis of DTD
DTD maintains robust VideoLLM performance under substantial token reduction, with feature-level dropping outperforming pixel-level dropping and training improving adaptation to extreme dropping ratios.
- Performance-Drop Ratio Tradeoff: At an 82.8% token-drop ratio, DTD achieves comparable performance to the full-token setting while maintaining robust results under substantial token reduction.Training with DTD improves adaptation to the dropped-token distribution, particularly at extreme dropping ratios of 80% or more.
- Performance-Drop Ratio Tradeoff: Feature-level dropping consistently outperforms pixel-level dropping.
6 Conclusion
TimeChat-Online introduces Differential Token Dropping for efficient, high-redundancy streaming video understanding. It achieves state-of-the-art streaming performance while eliminating up to 82.8% of visual tokens, with redundancy reduction especially valuable for hours-long videos.
- 6 Conclusion: TimeChat-Online proactively responds when future video moments answer a user’s question, triggering responses at scene transitions with low token-drop ratios.The case study illustrates proactive generation at future trigger timestamps indicated by frames with low token-drop ratios.
- 6 Conclusion: 82.8% of visual tokens are eliminated while TimeChat-Online achieves state-of-the-art performance on streaming video benchmarks.The conclusion reports that DTD removes up to 82.8% of visual tokens while preserving leading benchmark performance.
- 6 Conclusion: Over 95% of tokens can be eliminated from hours-long videos via DTD without performance degradation.The conclusion identifies temporal redundancy reduction as particularly critical for videos lasting several hours.
A Appendix
The appendix expands the paper with experimental results, training hyperparameters, dataset details, and visualization cases for Differential Token Drop.
- A.1 Experimental Results: Section A.1 reports extensive experimental results.
- A.2 Training Hyperparameters: Section A.2 presents the training hyperparameter tables.
- A.3 TimeChat-Online-139K: Section A.3 presents details of TimeChat-Online-139K.
- A.4 Differential Token Drop Visualizations: Section A.4 introduces diverse visualization cases for Differential Token Drop.
A.1 More Experimental Results
TimeChat-Online achieves state-of-the-art open-source performance on StreamingBench while retaining nearly all performance after substantial token reduction. Additional analyses show that DTD preserves essential spatiotemporal information and captures redundancy consistently across video content and tasks.
- StreamingBench Full-set Results: 58.11 overall StreamingBench score places TimeChat-Online 4.99 points above Dispider-7B, while Gemini 1.5 pro leads at 67.07.TimeChat-Online also surpasses Claude-3.5-Sonnet at 57.68 and all open-source offline VideoLLMs.
- StreamingBench Full-set Results: 82.6% token reduction retains 97.3% of original performance, with scores of 56.56 versus 58.11.This supports the conclusion that most visual tokens in streaming videos are redundant while essential information is preserved.
- Selection of Hyperparameters τ: Over 80% of visual information is naturally redundant across long-form videos, and τ_feat=0.25 is recommended for high performance with lower computational requirements.Consistency across datasets indicates that this redundancy is intrinsic regardless of content type or task domain.
- Breakdown Analysis of Fine-grained subtasks on MLVU: The overall improvement from 62.0 to 64.1 accompanies increases in the reported subtasks, including counting (AC) from 34.0 to 42.2.The supplied passage also reports another subtask increasing from 40.9 to 49.4, but its label is truncated.
- Breakdown Analysis of Fine-grained subtasks on MLVU: DTD preserves essential details despite dropping nearly 90% of tokens, supporting needle recognition, plot interpretation, temporal tracking, and accurate object counting.Its spatial-temporal aware design preserves the video’s original spatiotemporal structure after substantial token reduction.
A.2 Training Hyperparameter
This section indicates that TimeChat-Online’s training hyperparameters are documented in Table 7, but the supplied passage does not provide their values or settings.
- A.2 Training Hyperparameter: TimeChat-Online’s detailed training hyperparameters are provided in Table 7.The supplied passage references Table 7 but does not include its contents.
A.3 Dataset Statistics
The dataset contains 11,043 videos from 12 public source datasets, averaging 11.1 minutes each, and supports four high-level QA types spanning eleven fine-grained subcategories. Video-source composition, duration distributions, representative prompts, and construction templates are documented in referenced tables and figures.
- A.3 Dataset Statistics: 11,043 videos from 12 publicly available datasets average 11.1 minutes per video.Table 9 presents source composition, while Figure 5 shows video-duration distributions.
- A.3 Dataset Statistics: The QA pairs cover four types—Temporal-enhanced, Backward Tracing, Real-Time Visual Perception, and Forward Active Responding—across eleven fine-grained subcategories.
- A.3 Dataset Statistics: Representative prompts and streaming-QA construction templates are provided in Tables 10, 12, and 13.Table 10 covers representative examples; Tables 12 and 13 cover scene-oriented caption generation and streaming QA construction templates.
A.4 Diverse Visualization Cases
The visualizations show that DTD adaptively prunes temporally redundant tokens according to video content, while low drop-ratio valleys naturally identify scene transitions for proactive response generation.
- Token Dropping: DTD performs video-aware dynamic pruning: different frames and videos exhibit varying drop ratios, with diverse scenes less redundant than monotonous static scenes.These visualizations cover both feature-level and pixel-level token dropping under identical τ hyperparameters.
- Trigger Time: Low drop-ratio valleys reveal video scene transitions, enabling TimeChat-Online to monitor trigger times without additional perception modules.Figures 8 and 9 visualize trigger times using colored frames on the drop-ratio timeline, with temporal patch size 2.
- Hyperparameter Effects: Varying τ values produces different levels of feature-level and pixel-level redundancy control for the same video case.Figure 11 visualizes token dropping under different threshold values.
- Proactive Response: At future trigger times corresponding to scene transitions, TimeChat-Online proactively answers questions whose evidence appears in future moments.The case study asks what the woman in red did and shows proactive generation at future trigger timestamps marked by low token drop ratios.