Source-linked AI summary
MOSS-VL Technical Report
Pengyu Wang, Chenkun Tan, Shaojun Zhou, Qirui Zhou, Yanxin Chen, Xingyang He, Huazheng Zeng, Jijun Cheng, Chenghao Wang, Xiaomeng Qian, Pengfei Wang, Zhan Huang, Shanqing Gao, Wei Huang, Longjun Cao, Wu Ran, Jie Liu, Changtai Zhu, Hongkai Wang, Yixian Tian, Chenghao Liu, Zhen Ye, Xinghao Wang, Botian Jiang, Guoguo Feng, Zhaoye Fei, Ruixiao Li, Mingshu Chen, Yang Gao, Qinyuan Cheng, Shimin Li, Xipeng Qiu
TL;DR
Most open vision-language models analyze completed videos, whereas live assistants must perceive unfolding scenes and decide when to speak while continuing to watch. MOSS-VL co-designs gated visual interaction, synthesized timing supervision, and staged training; across streaming benchmarks, it leads three of four averages and scores 66.0 versus 37.5 on OmniMMI Proactive Alerting.
Problem
Most open vision-language models analyze completed clips, leaving live assistants to perceive unfolding scenes and decide when to speak while continuing to watch.
Method
MOSS-VL co-designs gated cross-attention, synthesized interaction data, and staged training to support perception during generation and response timing.
Results
66.0 vs. 37.5 on OmniMMI Proactive Alerting, with MOSS-VL-Realtime achieving the best average on three of four streaming benchmarks.
Takeaways & Limitations
MOSS-VL demonstrates an open vision-language model combining streaming perception, response timing, and generation, with strongest gains on timing-focused evaluations.
Takeaways & Limitations
Quantitative validation stops at L2–L4 because public streaming benchmarks do not measure perception during generation.
Abstract
from arXiv · showhide
We present MOSS-VL, an open vision-language model family that treats real-time interaction -- perceiving while it speaks -- as a first-class capability. It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention, so the model can naturally see incoming frames while generating; a synthesized interaction corpus supervises when to speak, when to stay silent, and when to revise; and a staged curriculum concentrates all real-time-specific training in one light final stage over a strong offline foundation. Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets. Across four streaming benchmarks, MOSS-VL-Realtime posts the best average on three (second on the fourth) among open-source streaming models, sweeping the three subsets that squarely test proactive behavior -- 66.0 vs. 37.5 for the best baseline on OmniMMI Proactive Alerting. With 11.3B parameters but visual tokens outside the decoded sequence, MOSS-VL widens its time-to-first-token advantage over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows. We release all five checkpoints, the training curriculum, and the real-time inference code at https://github.com/OpenMOSS/MOSS-VL.
1 Introduction
MOSS-VL treats real-time video interaction as a first-class capability, combining streaming-aware architecture, interaction supervision, and a lightweight final training stage. It achieves strong offline temporal reasoning and leading performance on proactive streaming evaluation while improving serving latency as visual context grows.
- MOSS-VL targets live video understanding, where assistants must decide when to speak and continue perceiving while generating.This extends beyond offline models that process completed clips before answering.
- Gated cross-attention keeps visual tokens outside the decoded sequence, allowing arriving frames to append to the cross-attention cache during generation.XRoPE and absolute timestamps support natural frame ingestion, while zero-initialized gates preserve the language backbone.
- Realtime-SFT uses synthesized interaction streams and under 3% of total training tokens to teach when to speak, stay silent, and revise.Two state tokens and one shared system prompt concentrate real-time-specific training in one light final stage.
- 66.0 vs. 37.5 on OmniMMI Proactive Alerting, while MOSS-VL-Realtime achieves the best average on three of four streaming benchmarks.It is second on the fourth benchmark and sweeps the three subsets testing proactive behavior.
- MOSS-VL-Instruct leads Minerva, TOMATO, and VideoMME-Logical, while MOSS-VL widens its time-to-first-token advantage from 2.8x to 5.1x as visual context grows.The latency comparison is against the same-backbone Qwen3-VL-8B and reflects visual tokens remaining outside the decoded sequence.
2 Architecture
MOSS-VL combines a native-resolution vision encoder with a Qwen3-8B-initialized decoder that accesses visual tokens only through gated cross-attention. Its XRoPE encoding and explicit timestamp tokens align visual and textual streams while preserving efficient incremental frame processing.
- Core architecture: Visual tokens reach the 48-layer decoder only through 12 tanh-gated cross-attention layers, while the other 36 self-attention layers operate on text alone.The vision encoder has 27 layers, and visual tokens remain outside the decoded sequence.
- Position encoding: XRoPE gives the cross-attention channel shared temporal, height, and width coordinates for text tokens and visual patches.Rotary frequency pairs are split (24, 20, 20) across (t, h, w), with rotations applied to text-side queries and vision-side keys.
- Temporal representation: Absolute timestamp tokens encode real time explicitly, keeping timing available across motion-adaptive video sampling from 1–16 fps.The positional t axis is a relative sequence coordinate rather than wall-clock time.
- Streaming inference: When a frame arrives, MOSS-VL encodes only that frame, appends its keys and values to the cross-attention cache, and preserves the existing decoding state.The decoded sequence grows only by timestamp tokens and one placeholder token, while patch tokens remain on the vision side.
3 Pre-Training
MOSS-VL uses a four-stage pre-training curriculum that progresses from connector alignment to broad and high-quality multimodal training, then long-context annealing. Large-scale synthesis of caption, OCR, grounding, and temporal-grounding data supports perception and temporal understanding throughout the curriculum.
- Curriculum: Four stages progress from vision–language alignment and large-scale multimodal pre-training to high-quality multimodal pre-training, then annealing with long-context training.The curriculum expands maximum sequence length from 8K to 256K tokens.
- Data synthesis: Large-scale synthesis supplies caption, OCR, grounding, and temporal-grounding data alongside reorganized existing corpora.These data types target scene description, embedded-text reading, object localization, and temporal event anchoring.
- Stage 1: Alignment: Stage 1 updates only the projection and cross-attention modules while keeping the vision encoder and language model frozen.Its image-captioning and OCR data use sequences capped at 8K tokens.
- Stage 2: Broad training: Stage 2 unfreezes the full model and broadens training across image, video, document, text-only, multimodal understanding, and reasoning data.The context window extends to 64K tokens for long interleaved documents and video.
- Stage 3: High-quality training: Stage 3 uses the curriculum’s largest token budget for higher-quality data, increasing multimodal understanding and reasoning while adding mathematics, knowledge-intensive, and temporal-grounding data.Sequences extend to 128K tokens.
- Stage 4: Annealing and long context: The final stage combines long-context training with high-quality annealing, including long-video and long-document data plus reweighted mathematics, knowledge-intensive, instruction-tuning, QA, and identity data.Long-context sequences stretch to 256K tokens.
4 Post-Training
Post-training uses two supervised stages: SFT creates an offline instruction follower, while Realtime-SFT adds frame-by-frame decisions to speak, remain silent, or revise. Its interaction-focused corpus and shared prompting support streaming behavior without architectural changes.
- Training stages: Two supervised stages transform MOSS-VL-Base into MOSS-VL-Instruct, then install real-time decisions without reinforcement learning or a thinking mode.Realtime-SFT teaches speaking, silence, and answer revision when the scene overturns an earlier response.
- Offline SFT: 7.6M instruction samples (102.8B tokens) train the offline model with next-token cross-entropy over assistant responses.Sequences extend up to 128K tokens, and the mixture combines collected, reorganized, and synthesized data after filtering and quality screening.
- Realtime-SFT corpus: 0.56M samples, ≈34.8B tokens comprise Realtime-SFT data centered on standing instructions, updating answers, commentary, accumulation, speaking timing, and video-independent interaction.Each sample casts the model in an explicit interaction role rather than a plain QA role.
- Data construction: Synthesized data specifically targets silence until evidence appears, answer revision as scenes evolve, and recovery after interruptions midway through replies.The corpus supplements strictly filtered and re-annotated open-source streaming-video datasets.
- Realtime-SFT corpus: 2.2M emission decisions train frame-by-frame interaction, with 58.7% self-timed decisions and 5.1% of samples requiring silence throughout when the target event never occurs.Streams run at 1 fps for up to 768 frames (≈12.8 minutes) per window.
- Mode control: One shared system prompt controls streaming and real-time modes, while offline inference uses none, enabling three modes with zero architecture change.Real-time operation is treated as a special case of streaming.
5 Infrastructure
MOSS-VL’s infrastructure combines scalable Megatron-LM parallelism and dense packing for long multimodal sequences with a custom FlashAttention-3 interface for streaming visibility patterns. Its serving stack is integrated with SGLang, while real-time inference is released separately through Transformers alongside the model weights.
- Training infrastructure: Megatron-LM combines data, tensor, sequence, and context parallelism to support 8K- to 256K-token curricula while densely packing variable-length image, video, and text samples.Mixed multimodal samples are packed into full sequences to keep batches dense.
- Attention infrastructure: Streaming cross-attention requires each text query to attend to the visual tokens of every preceding frame, creating a per-query visibility prefix that grows over time.This pattern is not directly served by off-the-shelf attention kernels.
- Attention infrastructure: MOSS-VL extends FlashAttention-3 with cross_kv_boundary, encoding each query’s visible prefix as one 32-bit integer across the operator, scheduler, and CUDA kernels.The interface avoids dense cross-attention masks whose memory and bandwidth cost scales with text-length times visual-length.
- Serving and release: The SGLang integration is upstreamed for offline serving, while real-time interaction is provided as a separate Transformers reference implementation with released model weights.The real-time implementation and weights are released on GitHub and HuggingFace.
6 Evaluation
MOSS-VL-Instruct is evaluated offline across 39 benchmarks, while MOSS-VL-Realtime is evaluated on four streaming benchmarks spanning capability levels L2–L4. The evaluation also measures serving efficiency and demonstrates real-time interaction behaviors in live sessions.
- Evaluation setup: The evaluation covers 39 offline benchmarks across five capability domains and four streaming benchmarks spanning levels L2–L4.Streaming evaluation feeds frames as they arrive under each model’s own streaming protocol.
- Offline evaluation: MOSS-VL-Instruct leads temporal-reasoning sets Minerva (40.5), TOMATO (39.5), and VideoMME-Logical (17.1), each by at least 4.9 points.It also takes five of twelve perception rows, including MMBench-EN (88.1), POPE (89.4), V* (89.0), MME-RealWorld (66.3), and BLINK (78.0).
- Streaming evaluation: MOSS-VL-Realtime posts the best average on OVO-Bench (70.2 vs. 65.3), OmniMMI (32.7 vs. 25.4), and ProactiveVideoQA (47.2 vs. 42.7), ranking second on StreamingBench (69.7 vs. AURA’s 71.1).These results compare against open-source streaming baselines across benchmark-specific subset panels.
- Streaming evaluation: 66.0 vs. 37.5 on OmniMMI Proactive Alerting, 60.0 vs. 53.2 on StreamingBench Proactive Output, and 62.1 vs. 55.8 on OVO-Bench Forward Active Responding show the strongest proactive gains.Backward Tracing on OVO-Bench is also 72.6 vs. 60.4, while AURA retains the edge on perception QA over the current scene.
- Serving efficiency: 2.8× to 5.1× is MOSS-VL’s time-to-first-token advantage over Qwen3-VL-8B as visual context grows, while end-to-end latency improves from 1.9× to 4.3×.The comparison uses matched ViT output, offline SGLang serving, and a single H200.
- Qualitative real-time sessions: In live sessions, MOSS-VL-Realtime stays silent except at four instructed contacts and begins commentary within a second while tracking a free-kick sequence through its outcome.The sessions exercise conditional response timing and continuous real-time commentary under real-world timing.
7 Discussion
The discussion attributes MOSS-VL’s evaluation profile to coordinated design across architecture, response-timing supervision, and curriculum, while identifying weaknesses in reasoning-heavy and document-centric tasks. It also emphasizes that real-time interaction remains incompletely evaluated and calls for reinforcement learning and an L5 benchmark.
- Discussion: MOSS-VL’s streaming gains, serving advantage, and offline temporal-reasoning strengths collectively reflect co-design across perception, generation, response timing, and curriculum.The evaluation pattern concentrates streaming wins on when-to-speak subsets, widens serving gains with visual history, and clusters offline strengths on temporal-reasoning video sets.
- Limitations: MOSS-VL-Instruct trails the strongest open models of its scale on reasoning-heavy suites such as MMMU and document-centric benchmarks.The report attributes these limitations to the absence of a thinking mode and training optimized for real-time video rather than exam-style reasoning.
- Limitations: Quantitative validation stops at L2–L4 because public streaming benchmarks stop there, leaving perception during generation without an existing benchmark.The discussion characterizes the highest-level real-time capability as still qualitatively attested.
- Future work: The near-term agenda includes reinforcement-learning post-training for MOSS-VL and a dedicated benchmark for L5 behavior.Reinforcement-learning post-training is already on the public roadmap, while the field still lacks an L5 benchmark.
8 Conclusion
MOSS-VL makes real-time interaction a first-class capability in an open vision-language model family through integrated streaming architecture, interaction supervision, and a staged curriculum. Its offline model retains strong performance, including on temporal-reasoning tasks.
- Core contribution: MOSS-VL makes real-time interaction a first-class capability of an open vision-language model family.The report frames this capability as built in rather than bolted on.
- Core contribution: Gated cross-attention with XRoPE lets frames arrive while text is being generated.This architecture supports perception during generation.
- Core contribution: Synthesized interaction data teaches the model when to speak, when to wait, and when to revise.The supervision targets the model’s timing and revision behavior during interaction.
- Training strategy: A staged curriculum confines every real-time-specific choice to one light final stage.The real-time-specific training is concentrated at the end of the curriculum.
- Evaluation: MOSS-VL-Instruct holds strong offline ground, especially on temporal-reasoning tasks.The passage describes strong offline performance but does not provide a numerical result here.
A Real-Time Interaction Details · A.1 Real-Time Dialogue Template
The real-time dialogue template represents streaming interaction as alternating decision slots and frame placeholders, enabling silence, incremental replies, reply closure, and repeated unprompted responses. It also standardizes timestamped frame expansion and runtime synchronization with training-time sequencing.
- A.1 Real-Time Dialogue Template: Streaming and real-time modes alternate assistant decision slots with frame placeholders, so N frames produce N+1 decisions.Offline inference instead encodes the full video as one vision block and uses ordinary chat completion.
- A.1 Real-Time Dialogue Template: Each decision slot is either <|silence|>, <|response|> plus a text chunk, or a text chunk closed by <|silence|>.The same silence token means idle silence when alone after a frame and reply completion when placed after text.
- A.1 Real-Time Dialogue Template: A session can begin with an empty user turn and silence in every slot, exercising silence-by-default before any instruction exists.A later turn can also close one reply, remain silent for a frame, and respond again when a second event occurs without a new user turn.
- A.1 Real-Time Dialogue Template: A reply spanning k frames emits k <|response|> tokens, one before each successive text chunk.This makes incremental speaking explicit rather than representing the whole reply with one response marker.
- A.1 Real-Time Dialogue Template: About half of training samples retain the leading slot before the next frame, while the rest drop it to handle questions arriving with or ahead of frames.The leading-slot augmentation covers both timing patterns equally well.
- A.1 Real-Time Dialogue Template: Each frame placeholder expands into a timestamped vision block whose plain-text arrival time is followed by resolution-dependent visual tokens.Training and inference must match this expansion byte for byte to avoid distribution shift and suppressed emission.
- A.1 Real-Time Dialogue Template: At inference, the runtime pushes one frame, waits for a fresh <|silence|>, and then pushes the next, while a length budget prevents long replies from stalling the stream.Replies therefore unfold across consecutive slots with one chunk per frame, matching training.
A.2 Live-Demo Session Transcripts
The live-demo section presents Chinese output excerpts from two sessions in Figure 5, with their English translations and stream placement provided in Table 7.
- Live-demo sessions: Figure 5 shows Chinese output excerpts from the model across two live-demo sessions.The excerpts are presented in the figure.
- Live-demo sessions: Table 7 provides English translations of both sessions in output order, while timestamps and stream placement come from Figure 5’s timeline.The translations are identified as the authors’ own.