Source-linked AI summary
VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding
Xinhao Li, Yuhan Zhu, Xiangyu Zeng, Yuhao Dong, Haoning Wu, Zhiqiu Zhang, Yuandong Yang, Changlian Ma, Qingyu Zhang, Yansong Shi, Xinyu Chen, Haoran Chen, Zizheng Huang, Jun Zhang, Kun Ouyang, Lin Sui, Ziang Yan, Yicheng Xu, Chenting Wang, Yinan He, Hongjie Zhang, Yi Wang, Yu Qiao, Yali Wang, Ziwei Liu, Kai Chen, Limin Wang
TL;DR
Open-source video models struggle with cross-domain generalization, computational efficiency, and complete openness. VideoChat3 combines efficient spatiotemporal architecture, scalable data synthesis, and full-stack release, achieving broad generalization and computational efficiency with a reproducible 4B-parameter model.
Problem
Existing open-source video models face limited cross-domain generalization, high computational overhead, and incomplete openness of training assets.
Method
VideoChat3 combines I3D-ViT, Adaptive Frame Resolution, scalable synthesis of three video datasets, and full-stack open-sourcing.
Results
VideoChat3 surpasses comparable-scale open-source video models while offering higher training throughput and inference speed with 4B parameters.
Takeaways & Limitations
VideoChat3 provides a fully reproducible, efficient foundation for general, long-form, and streaming video understanding research.
Takeaways & Limitations
Academic datasets are treated as reliable semantic anchors rather than definitive supervision targets.
Abstract
from arXiv · showhide
Recent advances in video understanding have spanned motion, long video, and streaming interaction, driving this field toward real-world applications. Despite this progress, current open-source models remain limited in several ways. They often struggle to generalize across diverse video types, making them effective only in specific domains. High computational demands further restrict their efficiency and scalability. Moreover, most models are only partially open, with key components such as training code, strategy, or datasets unavailable, which hinders reproducibility and slows community-driven development. To address these issues, we introduce VideoChat3, a fully open, efficient, and generalist video-centric MLLM. VideoChat3 advances video understanding through two complementary designs. For efficiency, we introduce Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for Streaming Video Perception, which enables efficient spatiotemporal representation and reduces the cost of processing video inputs during training and inference. For effectiveness, we develop a scalable video data synthesis pipeline that curates three diverse, high-quality training datasets: VideoChat3-Academic2M, VideoChat3-LV116K, and VideoChat3-OL617K, covering general, long-form, and streaming video scenarios, improving the model's generalization across domains. By integrating these designs, VideoChat3 achieves a rare balance of broad generalization and computational efficiency. Experiments across general, long-form, and streaming benchmarks demonstrate that VideoChat3 surpasses prior open-source models with equal or larger parameter counts with only 4B parameters and higher efficiency.
1 Introduction
VideoChat3 addresses limited cross-scenario generalization and efficiency in video MLLMs through an efficient architecture, scalable data construction, and full-stack open-sourcing. It targets broad video understanding across offline and streaming settings while improving reproducibility and computational efficiency.
- Motivation: Existing Video MLLMs often specialize in particular video settings, limiting transfer between short-form, long-form, offline, and streaming understanding.This restricts reliable deployment in real-world scenarios with varied video inputs.
- Overall Contribution: VideoChat3 combines efficient architecture, scalable data construction, and full-stack open-sourcing into a reproducible foundation for the open-source video modeling community.The framework lowers barriers to data, training, and efficient video modeling.
- Model Architecture: VideoChat3 introduces I3D-ViT and Adaptive Frame Resolution to reduce visual tokens and improve efficiency for high-frame-rate, long-form, and streaming videos.The design targets computational bottlenecks while maintaining strong understanding accuracy.
- Data Construction: The data pipeline produces VideoChat3-Academic2M, VideoChat3-LV116K, and VideoChat3-OL617K, collectively comprising 3 million high-quality multimodal instruction samples.These datasets span offline and online streaming video understanding tasks.
- Training, Evaluation, and Fully Open-source: VideoChat3 surpasses open-source state-of-the-art models of comparable parameter scale, including Qwen3-VL and Molmo2, while improving training throughput and inference speed.The claim is based on extensive evaluations across mainstream video understanding benchmarks.
2 Model Architecture
VideoChat3 addresses video-token inefficiency by modeling spatiotemporal structure and reducing redundancy before visual tokens reach the LLM. Its architecture combines I3D-ViT for local temporal modeling with Adaptive Frame Resolution for importance-dependent spatial processing, yielding efficient, detail-preserving understanding.
- Architecture overview: VideoChat3 compresses video representations along temporal and spatial dimensions before passing visual tokens to the LLM.This design targets the token-budget and compute limitations caused by increasing video duration, frame rate, and spatial resolution.
- Adaptive Frame Resolution: Adaptive Frame Resolution adjusts incoming-frame spatial resolution according to perceived importance, using low resolution for uninformative intervals and finer processing for potential events.In streaming video, state tokens control whether the next window uses a low or high pixel quota, enabling routine monitoring and focused inspection.
- I3D-ViT: I3D-ViT inflates a pretrained image tokenizer’s 2D spatial self-attention into chunk-wise 3D spatiotemporal attention over consecutive frames.It models neighboring frames jointly rather than encoding sampled frames as isolated images.
- I3D-ViT: Chunk-wise temporal pooling reduces token count by a factor of T after spatiotemporal contextualization.The pooled features already encode local temporal redundancy, avoiding frame-wise independent representations for subsequent modules.
- I3D-ViT: 16× spatiotemporal compression results from T = 4 temporal pooling combined with 2 × 2 spatial downsampling.The architecture also supports variable-length videos by adapting tokenization to the available visual budget.
3 Dataset Construction
VideoChat3 constructs supervision by enhancing reliable academic annotations, collecting and synthesizing long-video data, and converting QA pairs into streaming samples. The resulting datasets combine dense evidence-grounded responses with supervision for sparse, distributed temporal reasoning.
- Academic Source Aggregation: Academic datasets serve as reliable semantic anchors, providing complementary captioning, question-answering, and fine-grained motion supervision.Their original annotations are rewritten into denser responses while preserving the original answer as a semantic constraint.
- Evidence-Grounded Annotation Enhancement: Enhanced academic annotations expose visual evidence and reasoning implicit in labels without changing task definitions or adding unconstrained external knowledge.A verification stage uses Qwen3-VL-235B-A22B to check rewritten responses for semantic consistency with original annotations and filter unsupported details.
- Streaming Data Synthesis: High-quality video QA pairs are converted into online streaming samples so the model learns response content, timing, and proactive responding after sufficient visual evidence emerges.This extends the data construction process from answering questions to identifying when a response should be delivered.
- Long-Video Collection and Synthesis: Long-video construction addresses sparse evidence, delayed dependencies, event transitions, and cross-segment information aggregation through filtering, boundary-aware segmentation, and evidence-ledger annotation.Segments are annotated independently with entities, actions, scenes, transitions, temporal details, and OCR or subtitle cues before being assembled into full-video supervision.
- Long-Video Collection and Synthesis: 116.2K JSONL rows form VideoChat3-LV116K, whose long-video shards average 156s to about 1.3K seconds versus 3.0s to 59.0s for academic sources.The dataset provides supervision over much longer contexts where evidence is sparse and events are distributed across segments.
4 Training
Training proceeds through four stages, moving from language-grounded visual tokenizer pre-training to video-language alignment, general video instruction tuning, and long-form streaming adaptation. This staged recipe progressively builds visual representations, aligns them with the target language model, develops instruction following, and handles increasingly complex temporal requirements.
- Training Strategy: The four-stage schedule progresses from visual tokenizer pre-training through video-language alignment and general instruction tuning to long-form streaming adaptation.The stages are designed to increase temporal and instruction-following complexity over training.
- Stage-0: Stage-0 first warms up a projector with frozen ViT and LLM parameters, then jointly fine-tunes the ViT, projector, and LLM on a larger image-video mixture.The warm-up uses 1.04M samples, while full-parameter fine-tuning uses 6.55M samples; Stage-0 totals 7.59M single-turn samples.
- Stage-1: Stage-1 replaces the temporary decoder with the target language model and jointly aligns the visual tokenizer and LLM after projector warm-up.After this stage, images and videos become compact visual tokens that the LLM can reliably narrate.
- Stage-2: Stage-2 unfreezes all parameters and trains on balanced image-video QA and caption data, exposing the model to evidence selection, open-ended answering, and multiple-choice reasoning.The mixture contains 10.33M samples and 23.62M QAs, with image supervision anchoring fine-grained spatial perception and video supervision strengthening temporal reasoning.
- Streaming Adaptation: Streaming adaptation trains on progressively available video segments and requires answers based only on information observed so far.This encourages temporal memory, updating as new frames arrive, and retention of standard offline video performance.
5 Evaluation
VideoChat3-4B delivers strong performance across offline, online, and temporal-grounding evaluations while remaining efficient for long-video inference. Ablations further show that I3D-ViT and the proposed long-video and streaming data pipelines contribute substantially to these results.
- Offline evaluation: 61.7 on MotionBench and 75.6 on TempCompass are the best fully open results, demonstrating strong fine-grained motion and temporal-dynamics understanding.VideoChat3-4B improves on 18 of 19 directly comparable metrics against Qwen3-VL-4B, with the only decrease on VideoEval-Pro’s open-ended split.
- Offline evaluation: +16.4/+29.8/+34.3 on TimeLens and +30.6/+30.1 on VUE-TR show especially pronounced temporal-grounding gains over VideoChat-Flash-7B.VideoChat3-4B also matches or outperforms Molmo2-4B on most listed benchmarks and surpasses GPT-5 and Gemini 2.5 Flash on all three TimeLens splits.
- Online streaming evaluation: 72.3 on ODVBench surpasses StreamForest by 12.4 points, while VideoChat3-4B achieves the best results on four of six aggregate perception-and-memory metrics.It also exceeds the strongest competing results by 0.9 points on the OVOBench task average, 2.8 points on StreamingBench, and 2.9 points on River.
- Online streaming evaluation: 35.5 average F1 on OVO-Timing outperforms Em-Garde by 4.5 points without an auxiliary 2B module.Against Qwen3-VL-4B under the same settings, VideoChat3 improves on 10 of 11 directly comparable metrics.
- Efficiency analysis: At 2048 frames, total latency falls from 44.449s to 20.412s, while VideoChat3 also cuts FLOPs by more than 60% compared with Qwen3-VL.At 512 frames, FLOPs decrease from 1.341 to 0.864 ×10^15 and GPU memory from 32.92 GB to 26.68 GB; at 1024 frames, latency decreases from 12.251s to 8.099s.
- Ablation studies: OVO-Timing average F1 increases from 4.0 to 35.5 when VideoChat3-OL supervision is added, validating explicit timing targets for proactive streaming understanding.VideoChat3-LV116K boosts Video-MME, LongVideoBench, LVBench, ActivityNet, and QVHighlights, while online-data gains extend to streaming perception and memory with offline QA largely stable.
6 Related Work
Prior Video MLLMs extend image-based multimodal systems to dynamic video through visual encoders, projection modules, and language models, while recent work emphasizes efficient tokenization, scalable data, and temporal interaction. VideoChat3 complements these directions with early spatiotemporal compression using an inflated image ViT.
- Video Multimodal Large Language Models: Video MLLMs couple a visual encoder, projection module, and large language model to support open-ended video dialogue and temporal reasoning.Early systems including VideoChat, Video-ChatGPT, and Video-LLaVA demonstrated this video-centric instruction-tuning paradigm.
- Efficient Video Tokenization and Long-Video Modeling: Efficient video modeling reduces token redundancy through frame compression, spatiotemporal compression, KV sparsification, or input-side saliency mechanisms.Examples include VideoChat-Flash, LongVU, and Video-XL-2, alongside other approaches described in the passage.
- Efficient Video Tokenization and Long-Video Modeling: VideoChat3 instead inflates a pretrained image ViT into an I3D-ViT with chunk-wise spatiotemporal self-attention and temporal pooling before tokens enter the LLM.This design compresses local temporal redundancy early while preserving the visual representation for language modeling.
- Data-Centric and Open Video-Language Training: Video-language research increasingly relies on scalable pipelines that convert captions, question-answer pairs, or model-generated annotations into conversational supervision.Video-ChatGPT, VideoChat, ShareGPT4Video, and LLaVA-Video established representative data-construction pipelines, while LLaVA-OneVision-2 scales open supervision with re-captioned video and spatial corpora.
- Temporal Grounding and Streaming Video Understanding: Temporal grounding and streaming-video studies use timestamp tokens, boundary-aware objectives, or temporal streams to localize evidence and support interaction as videos unfold.Representative systems include TimeChat, VTimeLLM, LITA, VTG-LLM, and Grounded-VideoLLM.
7 Conclusion
VideoChat3 is presented as a fully open, efficient, and generalist Video MLLM addressing limited cross-domain generalization, high computational overhead, and incomplete openness of training assets. Its complementary efficiency- and effectiveness-oriented designs aim to balance broad generalization with computational efficiency.
- Conclusion: VideoChat3 is introduced as a fully open, efficient, and generalist Video MLLM.The model is framed as addressing three core limitations of existing open-source video models.
- Conclusion: The model targets limited cross-domain generalization, high computational overhead, and incomplete openness of training assets.These are identified as three core limitations of existing open-source video models.
- Conclusion: Two complementary designs targeting efficiency and effectiveness help VideoChat3 balance broad generalization with computational efficiency.The passage characterizes this balance as a central outcome of the proposed approach.
A Appendix · A.1 Implementation Details
Appendix A.1 specifies the evaluation configurations and reporting conventions for VideoChat3-4B across offline, online perception-and-memory, and online proactive-response video benchmarks. It also describes how configurations are applied to baselines and how unsupported or benchmark-specific evaluation requirements are handled.
- A.1 Implementation Details: Offline evaluations use benchmark-specific configurations covering sampling rate, maximum frames, per-frame pixel budget, and total visual-token budget.The same configurations are applied to baseline models whenever supported.
- A.1 Implementation Details: VideoChat3-4B and Qwen3-VL-4B are evaluated on online benchmarks using the configurations in Tables 11–12.Most remaining online-model results come from original papers; unavailable official results use a sampling rate of no more than 2 FPS within supported processing capacity.
- A.1 Implementation Details: Online perception-and-memory configurations specify sampling rate, maximum retained frames, per-frame pixel budget, and total visual-token budget.For OVOBench, 32 frames are processed in a sliding window rather than retained simultaneously.
- A.1 Implementation Details: ODVBench and OVBench evaluation changes the required bounding-box output range from [0, 1] to [0, 1000].This benchmark-specific modification is described alongside the online perception-and-memory evaluation settings.
- A.1 Implementation Details: Online proactive-response settings include a high-resolution window after the model predicts Standby.The window size is the number of consecutive frames processed at high resolution; Table 12 defines low- and high-resolution token budgets and subsequent high-budget frames.
- A.1 Implementation Details: OVOBench reports both overall and task-average scores, with “Task Avg.” averaging performance across subtasks within each benchmark component.This reporting convention applies to the perception-and-memory evaluation.
- A.1 Implementation Details: ProactiveVQA reports Proactive Area Under Curve (PAUC) using the default setting ω = 0.5.This convention is part of the online proactive-response evaluation reporting.
A.2 Demo
Qualitative demonstrations show VideoChat3’s broad video-understanding capabilities across motion, dense and long-video captioning, temporal reasoning and grounding, and online proactive response.
- Demonstration coverage: VideoChat3 covers fine-grained motion captioning, dense video captioning, long-video question answering, temporal reasoning, temporal grounding, and online proactive response.The demonstrations are presented in Figures 10–15.
- Fine-grained motion captioning: VideoChat3 describes a complete pole-vault sequence, including the run-up, pole planting, takeoff, bar clearance, and landing.
- Long-video question answering: VideoChat3 retrieves a specific visual detail from an extended video and correctly identifies the floor microphone stand’s color.
- Temporal reasoning and streaming response: VideoChat3 orders distributed events chronologically, localizes described events with only minor boundary deviation, and responds online only when informative visual content appears.
- Dense video captioning: VideoChat3 recognizes multiple events throughout a video and produces temporally coherent descriptions of its evolving content.