Source-linked AI summary
PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios
Xudong Lu, Huankang Guan, Yang Bo, Jinpeng Chen, Xintong Guo, Shuhan Li, Fang Liu, Peiwen Sun, Xueying Li, Wei Zhang, Xue Yang, Rui Liu, Hongsheng Li
TL;DR
Real-world mobile assistants need to reason over continuous audio-visual streams and respond at the right time, but existing benchmarks provide limited evidence for this setting. PhoStream introduces a long-form, open-ended mobile streaming benchmark with automated generation, temporal verification, online inference, and LLM-based judging. Models perform relatively well on Instant and Backward tasks but sharply worse on Forward tasks because early responses remain common.
Problem
Existing benchmarks lack specialized evaluation of mobile-centric streaming scenarios with diverse open-ended interactions and full temporal reasoning.
Method
PhoStream combines 5,572 open-ended QA pairs from 578 videos with automated generation, temporal verification, online inference, and LLM-as-a-Judge evaluation.
Results
Forward is the bottleneck: Gemini 3 Pro scores 80.83 on Instant and 82.19 on Backward, but 16.40 on Forward, with Early Response bias driving degradation.
Takeaways & Limitations
Current MLLMs remain unreliable at deciding when to respond in real-world streaming settings, not only what to say.
Takeaways & Limitations
Existing benchmarks lack specialized mobile-centric scenarios with open-ended model interactions, and StreamingVLM is omitted because it does not support open-ended question answering.
Abstract
from arXiv · showhide
Multimodal Large Language Models excel at offline audio-visual understanding, but their ability to serve as mobile assistants in continuous real-world streams remains underexplored. In daily phone use, mobile assistants must track streaming audio-visual inputs and respond at the right time, yet existing benchmarks are often restricted to multiple-choice questions or use shorter videos. In this paper, we introduce PhoStream, the first mobile-centric streaming benchmark that unifies on-screen and off-screen scenarios to evaluate video, audio, and temporal reasoning. PhoStream contains 5,572 open-ended QA pairs from 578 videos across 4 scenarios and 10 capabilities. We build it with an Automated Generative Pipeline backed by rigorous human verification, and evaluate models using a realistic Online Inference Pipeline and LLM-as-a-Judge evaluation for open-ended responses. Experiments reveal a temporal asymmetry in LLM-judged scores (0-100): models perform well on Instant and Backward tasks (Gemini 3 Pro exceeds 80), but drop sharply on Forward tasks (16.40), largely due to early responses before the required visual and audio cues appear. This highlights a fundamental limitation: current MLLMs struggle to decide when to speak, not just what to say. Code and datasets used in this work will be made publicly accessible at https://github.com/Lucky-Lance/PhoStream.
1. Introduction
PhoStream targets mobile streaming assistance, where models must integrate multimodal inputs and decide when to respond. It addresses benchmark gaps with long-form, open-ended evaluation and reveals severe early-response failures on Forward tasks.
- Motivation: Mobile assistants must integrate visual and audio streams while exercising temporal reasoning to decide when and how to respond.
- Benchmark gap: Existing streaming benchmarks often use multiple-choice questions, short clips, or narrow predefined tasks that do not represent diverse open-ended mobile interactions.
- Benchmark: PhoStream contains 5,572 open-ended QA pairs from 578 videos across 4 mobile-centric scenarios and 10 capabilities.
- Evaluation: Its evaluation combines an Online Inference Pipeline with LLM-as-a-Judge scoring for natural, well-timed, precise responses under streaming conditions.
- Findings: Models are too impatient on Forward tasks, often guessing before future evidence appears and struggling to decide when to speak rather than only what to say.
2. Related Works
Prior MLLM research emphasizes offline understanding and streaming efficiency, while existing benchmarks lack specialized mobile-centric evaluation with open-ended interactions. PhoStream addresses this gap through automated, temporally verified benchmark construction.
- MLLMs: Recent MLLMs demonstrate strong multimodal understanding, but many remain restricted to offline processing rather than real-time reasoning over continuous streams.
- Streaming systems: Streaming MLLM research improves memory, efficiency, asynchronous processing, or response decisions, but benchmark coverage remains limited.
- PhoStream: PhoStream uses an Automated Generative Pipeline to preprocess videos, generate candidate QA pairs, and apply multi-step verification with Gemini 3 Pro.
- Benchmark gap: Existing benchmarks primarily target general video understanding and lack mobile-centric scenarios with open-ended model interactions.
3. The PhoStream Benchmark
PhoStream is a mobile-centric benchmark built from long, multimodal streams and timestamped open-ended questions, with automated generation, human verification, and realistic online evaluation. Its design covers diverse scenarios and temporal task types while requiring models to decide when evidence is sufficient to respond.
- Dataset Construction: PhoStream combines on-screen and off-screen videos covering tutorials, app workflows, vlogs, travel, and first-person perspectives.The benchmark excludes external subtitles in curated online videos to evaluate visual and audio understanding.
- Task Definition: Online video QA requires answers to use only visual and audio evidence available before a strict timestamp cutoff.Backward questions use past context, Instant questions concern the current moment, and Forward questions become answerable only when later evidence appears.
- Annotation Pipeline: Gemini 3 Pro generates timestamped candidate QA pairs and verifies them against cutoff-specific video evidence before retention.The checker revises or discards samples that rely on future information, while 10 experts conduct two review rounds to correct, reject, or remove problematic samples.
- Quality Control: 5,572 final samples remain after automated filtering and human verification from 6,042 generated candidates.The automated check removes candidates violating streaming constraints or basic quality requirements before the final dataset is formed.
- Dataset Construction: 5,572 open-ended QA pairs from 578 videos form the final benchmark across four mobile streaming scenarios.The scenarios include YouTube Vlog, Phone Tutorial, Phone Record, and EgoBlind, with videos averaging 13.3 minutes.
- Online Evaluation: The online inference pipeline updates video and audio every second, exposes a 60-second sliding window, and queries each question only once.After querying, the model may answer immediately or continue monitoring until the required evidence appears; evaluation uses timestamp-based response windows and an LLM-as-a-Judge rubric.
4. Experiment
Experiments show a strong temporal asymmetry: models perform substantially better on Instant and Backward tasks than Forward tasks, where premature or missing responses are major failure modes. Audio improves some settings but can worsen Forward timing, while human evaluation reproduces the main trends.
- 4.2. Main Results: Models perform substantially better on Instant and Backward tasks than on Forward tasks.Proprietary models also achieve higher Overall scores than open-source models across settings.
- 4.2. Main Results: 80.83 on Instant and 82.19 on Backward contrast with 16.40 on Forward for Gemini 3 Pro.Qwen3-Omni similarly scores 77.18 and 77.24 on Instant and Backward, but 1.26 on Forward.
- 4.2. Main Results: 97.89% ER for Qwen3-Omni, 91.33% for Qwen3-VL-30B-A3B, and 79.12% for Gemini 3 Pro show that Early Response drives Forward degradation.Models often answer before sufficient cues are available, especially when optimized for streaming dialogue.
- 4.2. Main Results: Stronger Instant and Backward performance often coincides with more aggressive Forward answering and higher Early Response rates.Doubao-Seed-1.8 performs better than Doubao-Seed-1.6 on Instant and Backward but answers earlier and more frequently, contributing to a lower Forward score.
- 4.2. Main Results: 59.21% NR for MMDuet2 and 43.50% for Qwen2.5-Omni-7B indicate a distinct Forward failure mode involving missed response windows.This reflects a response-triggering bottleneck rather than premature answering.
- 4.3. Audio Ablation Analysis: Adding audio improves Instant, Backward, and Overall performance but can slightly reduce Forward scores and increase ER.The ablation indicates that richer multimodal signals may encourage responses before the proactive timestamp.
- 4.4. Human Test: Human annotations on 200 sampled videos reproduce the main evaluation trends, including open-source models lagging on Forward tasks.The Forward gap is associated largely with excessive early responses or missing outputs.
5. Conclusion
PhoStream is a mobile-centric benchmark for evaluating omnimodal assistants across on-screen and off-screen streaming scenarios. Its results show severe Early Response bias on Forward questions, indicating that current MLLMs remain unreliable at deciding when to respond.
- PhoStream unifies on-screen and off-screen scenarios in a mobile-centric benchmark for omnimodal assistants.
- Current MLLMs show severe Early Response bias on Forward questions and remain unreliable at deciding when to respond in streaming settings.
Impact Statements
PhoStream evaluates streaming-video understanding in mobile-centric scenarios and identifies premature answering as a critical weakness of current models.
- Current models tend to answer before observing necessary visual and audio cues from the video stream.
A.1. More Statistics of PhoStream
PhoStream questions cover diverse mobile intents and span timestamps across long videos. The mean query timestamp is 6.52 minutes, with a maximum of 30.87 minutes.
- The 5,572 open-ended questions cover diverse user intents and topics in mobile scenarios.
- 6.52 minutes is the mean query timestamp, while 30.87 minutes is the maximum.The wide temporal range challenges models to maintain long-term context.
A.2. Details of Human Annotators
The benchmark and human evaluation stages used 10 expert annotators with multimodal video-understanding experience and at least master’s degrees. They were paid $20 per hour.
- 10 expert annotators supported both benchmark annotation and human testing.The annotators had experience in multimodal video understanding.
- Each annotator held at least a master’s degree.
- Annotators were compensated at $20 per hour.
A.3. More Detailed Evaluation Results
The appendix provides detailed evaluation breakdowns across dataset categories and capabilities for Instant, Backward, and Forward tasks. Results identify the best-performing model in each reported category.
- Dataset breakdown: Evaluation scores are broken down across YouTube Vlog, Phone Tutorial, Phone Record, and EgoBlind datasets.These results are summarized in Table A.1.
- Task settings: The detailed results cover Instant, Backward, and Forward task settings.The settings are reported for both dataset and capability subsets.
- Capability breakdown: Capability-level tables report results across ten capability categories, including UI & Digital Interface and Visual Text Understanding.Tables A.2–A.4 provide the capability breakdown, with the best result in each category bolded.
A.4. Failure Cases and Qualitative Results on PhoStream
The qualitative cases show that streaming QA depends on responding only after the necessary evidence appears and on correctly integrating audio, visual, and whole-stream information. Models frequently answer too early, fail to answer, or misinterpret evidence even when responding at the required time.
- Timing failures: Many models answer before the required visual evidence appears, making response timing a key factor in correctness.The evaluation records each model’s first substantive response under strict timing cutoffs.
- Timing failures: Most models answer the New York Giants page question at 11:30, before the relevant content becomes visible at 11:40.Doubao-Seed-1.6 responds at 11:40 and is partly correct, while Gemini 3 Pro is labeled EarlyResponse.
- Timing failures: Several models answer the Roll-Up Challenge question immediately, although the winner is only declared at 10:38.Christian wins with 35 roll-ups versus Charles’s 28; some models are EarlyResponse and others are NoResponse.
- Audio understanding: When the required evidence is in the audio, models can still fail by missing or misinterpreting the voiceover.Gemini 3 Pro captures the mobile-data complaint correctly, whereas Qwen3-Omni only partially matches it and other models guess or fail.
- Long-range aggregation: Whole-stream aggregation remains challenging even when models respond at the correct time.Parked cars are the dominant obstacle, but some models instead focus on salient background objects such as trees.
- Long-range aggregation: Some models identify parked cars as the most frequent large obstacle, while others answer trees and receive lower scores.The walk repeatedly includes parked cars obstructing pedestrian paths.
A.5. Placeholder Responses for Filtering in Evaluation
The evaluation filters generic placeholder or silent responses before scoring. It then uses an LLM judge to assess whether remaining answers provide reasonable, relevant, and factually plausible explanations.
- Filtering: Placeholder filtering removes empty, silent, no-information, reminder-like, and other generic acknowledgment responses.The defined placeholder set includes English and Chinese variants.
- LLM evaluation: The LLM-as-a-Judge prompt evaluates whether a model answer directly addresses the question with a reasonable and factually plausible explanation.The judgment is based on comparison with the reference answer.
- LLM evaluation: The judge prioritizes coherent causal reasoning and accepts concise or rephrased answers when they preserve the essential logic.Secondary or illustrative details are not required when the core reasoning is present.
- Scoring: Answers receive integer scores from 0 to 5, ranging from fully accurate and complete to no attempt or completely off-topic.