PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding

Peiwen Sun, Xudong Lu, Huadai Liu, Yang Bo, Dongming Wu, Huankang Guan, Minghong Cai, Jinpeng Chen, Xintong Guo, Shuhan Li, Fang Liu, Rui Liu, Xiangyu Yue

arXiv:2606.02482v3cs.CV

TL;DR

Existing benchmarks do not adequately evaluate continuous cross-stream reasoning, despite its importance for real-world multi-stream applications. X-Stream addresses this gap with a curated benchmark and dual-verified data pipeline, while framing MLLMs as naive multiplexers; experiments show top-performing models achieve only about 50% score and current multiplexing strategies involve trade-offs.

  • Problem

    Existing multi-video datasets typically lack streaming characteristics and long-duration, accurately timestamped multi-stream annotations, limiting evaluation of continuous cross-stream understanding.

  • Method

    X-Stream combines 4,220 curated QA pairs across 932 videos with a dual-verification pipeline and evaluates MLLMs as naive multiplexers under online streaming inference.

  • Results

    Most top-performing models achieve only about 50% score, while multiplexing effectiveness depends on token bandwidth and concurrent stream count, producing strategy trade-offs.

  • Takeaways & Limitations

    X-Stream provides a practical benchmark for multi-stream streaming understanding and shows that advanced cross-stream skills, including causal reasoning, remain far from application.

  • Takeaways & Limitations

    The data pipeline must address pseudo multi-stream questions whose temporal references or redundant views allow answers from a single stream.

Abstract

from arXiv · show

While video streaming understanding has made significant strides, real-world applications, such as live sports broadcasting, autonomous driving, and multi-screen collaboration, inherently demand continuous, multi-stream interactions. However, existing benchmarks are confined to single-stream paradigms, leaving a critical gap in evaluating online, cross-stream reasoning. To bridge this, we introduce X-Stream, the first benchmark dedicated to multi-stream streaming understanding. Comprising 4,220 rigorously curated QA pairs across 932 videos, X-Stream evaluates 11 subtasks across multi-window, multi-view, and multi-device scenarios. Crucially, our dataset is constructed using a novel dual-verification pipeline that prevents over-reliance on a single stream. Furthermore, we pioneer the conceptualization of multi-modal large language models (MLLMs) as naive multiplexers, systematically evaluating their performance through the lens of Signal Multiplexing Theory. Our extensive online inference experiments reveal a stark reality: state-of-the-art MLLMs struggle significantly with concurrent streams, achieving only about 50% score and exhibiting poor proactive ability. Ultimately, X-Stream exposes the trade-off of current multiplexing schemes, providing both a practical evaluation protocol and empirical guidance for next-generation multi-stream agents.

1 Introduction

X-Stream addresses the need for online understanding across synchronized video streams, where existing benchmarks lack streaming characteristics and timestamped multi-stream annotations. It introduces a benchmark and multiplexing-based evaluation showing that current MLLMs remain limited on concurrent streams.

  • Motivation: Real-world applications require continuous collaboration across multiple simultaneous video streams, including sports broadcasting, office coordination, navigation, and robotics.These settings involve coordinated feeds and temporally connected observations.
  • Research gap: Existing multi-video datasets typically lack streaming characteristics and long-duration, accurately timestamped multi-stream annotations.X-Stream is designed to address this evaluation gap.
  • Benchmark: X-Stream evaluates 4,220 QA pairs across 932 videos and 451 takes, covering 4 multi-stream capabilities, 3 progressive dimensions, and 11 subtasks.The tasks span multimodal perception, logical cognition, and decision-making, requiring continuous integration of multi-stream cues.
  • Multiplexing formulation: MLLMs are conceptualized as naive multiplexers that integrate multiple video streams into one token stream under bounded token-processing capacity.The framing applies the signal-multiplexing concept to concurrent video inputs.
  • Multiplexing strategies: Spatial, temporal, and semantic division strategies exhibit trade-offs that depend on token bandwidth and the number of concurrent streams.Spatial division helps cross-stream referencing, whereas semantic division becomes more necessary for three or more streams under tight token budgets.
  • Findings: Most top-performing models achieve only about 50% score, while advanced cross-stream skills such as causal reasoning remain far from application.The benchmark therefore exposes substantial limitations in current multi-stream understanding.

2 Related Works

Related work has advanced single-stream streaming, multi-video, multi-view, and multi-angle understanding, but online multi-stream reasoning remains unexplored. The defining gap is the combination of simultaneous streams with timestamp alignment and real-time evaluation.

  • Streaming understanding: Recent streaming-video research incrementally processes incoming text and frames to support real-time understanding and interaction for long-form single streams.Existing efforts address memory, processing efficiency, and interaction in streaming dialogues.
  • Multi-video taxonomy: The multi-video data family can be organized as a pyramid of increasing constraints: Multi-Video, Multi-Stream, Multi-View, and Multi-Angle.This taxonomy distinguishes broad multi-video understanding from more constrained synchronized or perspective-specific settings.
  • Multi-stream distinction: Multi-Stream introduces timestamp alignment, whereas prior Multi-View and Multi-Angle approaches mainly analyze complete multi-view files rather than online real-time streams.Timestamp synchronization is the critical constraint separating multi-stream understanding in this hierarchy.
  • Open problem: Multi-stream streaming understanding remains unexplored because prior approaches do not evaluate synchronized multi-stream inputs in an online or real-time manner.This motivates a benchmark dedicated to the combined setting.

3 Data Construction

X-Stream constructs a diverse, timestamped multi-stream benchmark and organizes evaluation around temporal requirements and cross-stream abilities. Its pipeline combines automated generation with sufficiency, necessity, and human verification to prevent single-stream shortcuts.

  • Data collection: The benchmark construction protocol covers data collection, task definition, annotation, and statistical analysis across multi-angle, multi-view, and multi-device configurations.The collection includes 857 hours of raw data with 2 to 10 concurrent streams from more than 20 sources.
  • Temporal task design: Queries are categorized as instant or forward questions according to whether answers can use retrospective/current context or must be produced proactively at the correct future time.This categorization captures the temporal requirements of online streaming understanding.
  • Multi-stream abilities: The framework defines 4 core abilities: single-stream understanding, cross-stream anti-interference, cross-stream reference alignment, and cross-stream cooperation.These abilities range from extracting stream-specific information to synthesizing clues unavailable from any single stream.
  • Task taxonomy: X-Stream evaluates 11 progressive tasks spanning multimodal perception, logical cognition, and higher-level reasoning.The task inventory includes grounding, counting, saliency detection, spatial, causal, counterfactual, commonsense, and anomaly reasoning.
  • QA pipeline: QA construction uses preprocessing, QA generation, sufficiency and necessity verification, and human verification.Generation combines MLLM-based rejection sampling with templates and timestamped metadata.
  • Quality control: Dual verification removes pseudo-reference and pseudo-cooperation shortcuts by requiring multi-stream sufficiency while rejecting questions answerable from isolated streams.Human experts additionally review question clarity and answer accuracy, with temporal criteria for instant and forward questions.

4 MLLMs as Naive Multiplexers

The paper frames MLLMs as naive multiplexers that merge concurrent video streams into one token stream under bounded token bandwidth. It evaluates spatial, temporal, and semantic division strategies, whose trade-offs depend on stream count and available capacity.

  • Conceptualization: Multiplexing combines multiple signals into one signal over a limited shared medium, motivating analogous integration of video streams for MLLMs.MLLMs process one token stream at a time, so multiplexing provides a lens for analyzing concurrent-stream processing.
  • Conceptualization: A fixed average video token rate Cmax models the context-window and computational limits of practical multi-stream inference.The benchmark enforces this bandwidth-like constraint throughout evaluation.
  • Multiplexing strategies: Spatial division stitches downsampled frames from multiple streams into one input subject to |T(Xt)| ≤ Cmax.It uses pixel-level concatenation after spatial downsampling, but requires video re-encoding and overlapping audio beforehand.
  • Multiplexing strategies: Temporal division samples frames from separate streams under a token budget and aligns their temporal embeddings using stream identifiers.Synchronized temporal encoding is feasible only with open-source models in practice.
  • Multiplexing strategies: Semantic division selects relevant and visually diverse tokens using relevance-similarity scores and greedy MAP inference before interleaving streams.The selection penalizes candidates similar to already selected tokens, producing compact cross-stream token subsets.
  • Unified processing: All multiplexing schemes integrate multiple videos into a unified token sequence before MLLM inference.The resulting sequence is passed to VLLM for online inference.

5 Experiments

Experiments evaluate streaming and multi-stream abilities under controlled online inference, then compare model capability hierarchies and multiplexing behavior. Results show strong foundational performance but substantial weaknesses in advanced reasoning, proactive timing, and scaling across streams.

  • 5.1 Experiment Setup: The evaluation uses a 2-second response window across 6 time slots, with temporally accurate answers averaged for multi-stream ability scores.The average token budget is capped at Cmax = 250 tokens per video second, enforced through model-specific playback or resizing adjustments.
  • 5.2 Main Results: Proprietary models consistently outperform open-source counterparts across all streaming settings on the full X-Stream benchmark.Qwen3-Omni-30B-A3B leads open-source models, but its Forward capability is limited by response timing.
  • 5.2 Main Results: Multi-stream capability declines from foundational perception to logical cognition and decision-making, with causal reasoning among the strongest bottlenecks.Dimension scores exclude timing effects, producing higher overall scores than timing-sensitive streaming evaluation.
  • Multiplexing analysis: Under many concurrent streams, spatial and temporal division lose usable information through blurriness or discontinuity, whereas semantic division retains basic semantic content.For a single stream, spatial and temporal division preserve standard streaming video while semantic division introduces unnecessary semantic loss.
  • Multiplexing analysis: Spatial division excels at temporal modeling and cross-stream referencing, while multiplexing effectiveness varies with stream count, bandwidth, and task demands.No single multiplexing approach is universally optimal across constraints.
  • Audio analysis: Audio-capable methods perform significantly better on audio grounding, but spatial division can overlap channels and time division can cause speech discontinuity.Because audio signals are more strongly coupled than image pixels, the analysis remains limited to simple multiplexing techniques.
  • Necessity of multi-stream: Single-stream inference fails on the multi-stream benchmark, while adding distractor streams to single-stream datasets causes severe performance degradation.These results support X-Stream as a test of a distinct multi-stream capability rather than a simple extension of single-stream systems.
  • Evaluation validation: LLM-as-a-Judge agrees with human evaluation at Spearman correlation 0.62 (p<0.05) over 200 QA pairs.The human verification supports the reliability of the evaluation procedure.

6 Discussion and Conclusion

X-Stream targets the underdeveloped problem of continuous multi-stream understanding with a benchmark designed to require cross-stream reasoning. Its experiments show that current MLLMs still struggle with concurrent integration and proactive tasks, while multiplexing strategies involve trade-offs.

  • Conclusion: X-Stream is presented as the first comprehensive benchmark for multi-stream streaming understanding, with 4,220 curated QA pairs requiring cross-stream understanding.The benchmark uses a rigorous dual-verification pipeline to ensure that its questions genuinely demand multiple streams.
  • Conclusion: Current MLLMs achieve only around 50% accuracy and fall short on proactive tasks when integrating continuous multi-stream inputs.The conclusion characterizes these models as naive multiplexers and reports difficulty with concurrent stream integration.
  • Discussion: Public video datasets are constrained by stream drift over time because precise synchronization typically requires professional equipment and expert work.This limitation is identified on the data side of multi-stream understanding.
  • Discussion: Existing multiplexing strategies struggle to balance video comprehension with temporal reasoning, and their trade-offs vary with token constraints and stream counts.The paper analyzes these trade-offs across three multiplexing strategies.

A Multi-Stream Data Preview

Multi-stream perception is framed as a broad real-world requirement spanning heterogeneous devices and coordinated views. The data preview highlights this application diversity across settings such as offices, sports, navigation, and robotics.

  • Applications: Multi-stream collaboration supports multi-screen offices, live sports feeds, cooperative map-and-glasses navigation, and synchronized robotic-arm cameras.These examples involve heterogeneous devices and coordinated streams.
  • Data Preview: The X-Stream data preview highlights real-world multi-stream applications and the diversity of the benchmark.The preview is intended to provide an overview of the dataset’s application coverage.

B.1 Data Sources

X-Stream combines diverse multi-domain sources with scenario-specific tasks that test how streams are combined, contrasted, referenced, and used for shared goals. Its examples span different views, devices, behaviors, and application settings.

  • Data Sources: Approximately 857 hours from 20 methods and sources were screened to retain about 160 hours of final data.These statistics summarize the multi-domain source collection and final benchmark inputs.
  • Data Sources: Reaction videos use face-camera and screen streams to capture real-time responses to shared video or gameplay content.The source collection includes clips from ten popular streamers.
  • Data Sources: Gameplay sources include competitive esports titles and other cooperative games, with gameplay-only recordings serving as standardized base streams.The selected titles cover different interaction and viewing dynamics.
  • Data Sources: Controlled capture with HLAE and OBS provides multiple gameplay viewpoints when suitable public multi-view streams are unavailable.The pipeline programmatically controls playback and records the resulting perspectives.
  • Data Sources: Driving data synchronizes video and telemetry by using CAN speed timestamps as the reference axis and interpolating other sensor signals onto it.The paired dataset includes signals such as steering angle and wheel speed.
  • Task Taxonomy: The task taxonomy covers cross-stream interference, cooperation, reference, and single-stream understanding as ways heterogeneous streams contribute to answers.The taxonomy organizes how information is combined, contrasted, and linked.
  • Scenario-Specific Tasks: Scenario-specific QA covers multi-angle observation, multi-view behavior understanding, and multi-device collaboration for a shared goal.Examples include autonomous driving, collaborative gaming, social interaction, robotics, street-view and map pairing, and sports analysis.
  • Scenario-Specific Tasks: Examples connect paired streams to tasks such as manipulation-failure diagnosis, ingredient reference resolution, rule verification, and cross-camera reidentification.The examples illustrate questions that require complementary evidence across views or devices.

B.4 Dataset Statistics

The benchmark preserves broad variation in its original data and produces a relatively balanced final distribution across question formats and answer types. Human review further quantifies correction needs and the distribution of annotation errors.

  • Original Data: The original data vary substantially in duration, video count, stream type, domain consistency, and stream count.The collection emphasized broad distribution coverage and diversity aligned with real-world scenarios.
  • Final Dataset: The processed dataset has relatively balanced question categories and diverse free-form answer lengths alongside multiple-choice options.This combines structured and open-ended formats for model evaluation.
  • Human Annotation: 31 expert annotators with multimodal video-understanding experience supported benchmark annotation and human testing.Annotators were compensated at $18 per hour.
  • Human Annotation: Human reviewers checked synchronized multi-stream context, timestamped questions, pre-filled answers, and explanations under a structured quality-control protocol.Judgments were grounded in visible video evidence at the specified temporal segment, with revisions allowed for temporal, semantic, or linguistic issues.
  • Human Correction Statistics: 25.6% of evaluation cases required manual correction, and accuracy after correction reached 94.5%.Among corrected errors, question errors were 65.2%, answer errors 21.7%, and timestamp errors 13.0%.
  • Licensing: The dataset uses CC BY 4.0 or Apache License 2.0 for different source-data groups after reviewing open-source licenses.These licenses were among the most widely adopted in the reviewed sources.

C Experiment Detail

The experiments face token-budget limitations because visual processors tokenize video differently, requiring model-specific compromises in speed and resolution. Temporal QA can also appear strong through single-stream shortcuts rather than genuine cross-stream grounding.

  • Different visual tokenization mechanisms make token counts vary across models, requiring speed and resizing compromises for baseline control.Gemini uses fixed tokens per second, Qwen3-VL uses patch tokens with merging, and GPT-5 combines frame and tile tokens.
  • Strong temporal QA performance does not necessarily indicate genuine temporal grounding because persistent context and redundant cues can support shortcut answers.Questions involving state changes, causal reactions, or transient attributes require precise cross-stream alignment.

D.2 Analysis of Grid Layout and Spatial Division

Grid layout affects spatial multiplexing by changing raster-scan token interleaving and stream separability. Vertical stitching generally performs better because it keeps streams more distinct during attention.

  • Analysis of Grid Layout and Spatial Division: Vertical-level stitching generally performs better than horizontal-level stitching in spatial division multiplexing.The authors attribute the gap primarily to raster order’s influence on the attention mechanism.
  • Analysis of Grid Layout and Spatial Division: Invalid temporal anchoring occurs when a target action is continuous or static, making the cross-stream timestamp constraint redundant.If the answer is invariant over time, the reference-stream event does not meaningfully determine the answer.
  • Analysis of Grid Layout and Spatial Division: Raster scanning interleaves tokens from multiple streams under horizontal concatenation but generally prevents such interleaving under vertical concatenation.The scan proceeds left-to-right and top-to-bottom, so layout changes the order in which stream tokens enter the model.
  • Analysis of Grid Layout and Spatial Division: Vertical stitching keeps streams more clearly separable, reducing misinterpretation caused by token interleaving and supporting coherent global understanding.

D.3 Analysis of Temporal Embedding for Time Division

Time Division multiplexing requires temporal embeddings that distinguish concurrent streams’ moments. Assigning identical timestamps can erase this distinction and substantially degrade performance.

  • Analysis of Temporal Embedding for Time Division: Time Division assigns concurrent frames from different streams to the same temporal embedding, requiring continuous timestamps to preserve their temporal distinction.
  • Analysis of Temporal Embedding for Time Division: Identical timestamps can make distinct moments indistinguishable, causing performance degradation of up to 30% and loss of core capabilities.
  • Analysis of Temporal Embedding for Time Division: Raster-scan order remains relevant to how multiplexed video tokens are arranged and processed, even when streams are combined across dimensions.
Loading 2606.02482v3…