Source-linked AI summary
FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?
Yuxuan Hu, Weikang Shi, Yang Bo, Xudong Lu, Xintong Guo, Shuhan Li, Yuyang He, Huankang Guan, Peiwen Sun, Yunqiao Yang, Wenbo Li, Rui Liu, Hongsheng Li
TL;DR
Streaming VLMs face a gap in evaluating high-dynamic perception under bounded context budgets. FastBench combines low-FPS filtering, trajectory-grounded verification, and human review, while ProactiveFrame adjusts temporal sampling. The best model reaches only 50.7%, and deciding when to look closer remains an open challenge.
Problem
Existing streaming benchmarks do not explicitly target high-dynamic perception, which requires fine temporal detail alongside long history under limited context budgets.
Source: §1 · p. 2, §3.1 · p. 5Method
FastBench filters questions answerable at 2 FPS and verifies retained answers against SAM3 and CoTracker3 trajectories, while ProactiveFrame adjusts frame rates and manages context with a dual-tier window.
Source: §1 · p. 2, §3.1 · p. 5Results
The best model reaches only 50.7% on FastBench, and ProactiveFrame improves over sparse sampling yet trails oracle-guided focusing.
Source: §5 · p. 10Takeaways & Limitations
Deciding when to look closer remains the central open challenge for proactive temporal control in streaming video understanding.
Source: §5 · p. 10ProactiveFrame's gap to oracle-guided sampling highlights the challenge of autonomously identifying when finer temporal perception is needed.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.
1 INTRODUCTION
Streaming VLMs must interpret continuous video under tight context budgets, but existing streaming benchmarks do not explicitly target high-dynamic perception. FastBench evaluates that gap, alongside ProactiveFrame, a training-free approach to adjusting temporal sampling.
1 INTRODUCTION
- High-dynamic streaming requires balancing long temporal history, spatial resolution, and fine-grained temporal information under limited token budgets, while sparse 1–2 FPS sampling can miss crucial details.Existing streaming benchmarks emphasize temporal reasoning and response timing without explicitly targeting high-dynamic perception; continuous high-dynamic evaluation remains open.Source: §1 · p. 2
- FastBench evaluates streaming VLM perception of high-dynamic real-world streams using low-FPS filtering, trajectory-grounded verification, and human inspection.Its pipeline filters pseudo-dynamic questions and grounds answer verification in physical trajectories.Source: §1 · pp. 2–3
- ProactiveFrame uses text prompts to adjust future frame rates and a dual-tier window to retain recent high-FPS chunks alongside sparse historical context.The window lets focused chunks degrade into sparse history under strict token budgets.Source: §1 · pp. 2–3
- ProactiveFrame improves over sparse uniform sampling, but its gap to oracle-guided sampling highlights the difficulty of autonomously identifying when finer temporal perception is needed.The paper identifies this autonomous focusing challenge as a limitation of current VLMs.Source: §1 · pp. 2–3
2 RELATED WORK
Prior streaming benchmarks assess temporal understanding and response timing, while fine-grained motion benchmarks study motion or frame-rate requirements in offline settings. ProactiveFrame combines prompt-driven temporal sampling with dual-tier context management for continuous streams.
2.1 STREAMING VIDEO UNDERSTANDING BENCHMARKS
- StreamingBench, OVO-Bench, OVBench, and PhoStream assess streaming understanding across temporal tasks, but FastBench targets high-frame-rate-dependent evidence under bounded visual context.FastBench jointly evaluates fine-grained perception, historical retention, and response timing.Source: §2.1 · p. 3
2.2 FINE-GRAINED MOTION AND HIGH-FRAME-RATE UNDERSTANDING
- TempCompass, MotionBench, and FPS-Bench address motion comprehension or frame-rate needs, while FastBench uses 2 FPS answerability filtering in a causal streaming protocol.Unlike FPS-Bench's per-question minimum-FPS estimation, FastBench filters questions answerable from low-FPS samples.Source: §2.2 · p. 4
2.3 PROACTIVE PERCEPTION AND STREAMING CONTEXT MANAGEMENT
- ProactiveFrame differs from prior streaming approaches by adjusting future temporal sampling through prompts while retaining recent high-FPS observations and sparse historical context.Other cited approaches address response determination, event-gated invocation, resolution adaptation, or context retention.Source: §2.3 · p. 4
3 FASTBENCH
FastBench is a streaming QA benchmark built from high-dynamic video, with evidence intervals and three temporal scopes. ProactiveFrame offers a training-free way to adjust sampling while retaining a sparse history.
3 FASTBENCH
- FastBench formulates streaming QA over continuous video, with questions requiring backward, instant, or forward temporal evidence.The benchmark targets fine temporal granularity and the trade-off between historical context and high-frequency visual detail under limited context budgets.Source: §3.1 · p. 5
3.1 DATA CURATION
- FastBench curation generates QA pairs from high-FPS clips, drops questions answerable at 2 FPS, verifies answers against SAM3 segmentation and CoTracker3 trajectories, and subjects retained pairs to three expert-inspection rounds.Experts check correctness, clarity, and dependence on high-dynamic visual information, then annotate evidence intervals and high-FPS timestamps.Source: §3.1 · p. 5
3.2 DATA STATISTICS
- 306 high-quality QA pairs remained from 16,702 candidates after low-FPS filtering, trajectory verification, and multi-round human inspection.The dataset spans six capability dimensions and eight high-dynamic domains.Source: §3.2 · pp. 6–7
3.3 PROACTIVE FRAME
- ProactiveFrame lets streaming VLMs switch incoming frame rates through prompts, addressing the trade-off between temporal detail and long-term history.With a fixed 120-frame context, 2 FPS preserves 60 seconds while 30 FPS retains only 4 seconds; the focus window buffers high-FPS chunks, which are downsampled into sparse history when displaced.Source: Figure 4 · p. 6, §3.3 · p. 7
4 EXPERIMENT
FastBench evaluation shows that proactive sampling improves performance, but dense temporal perception remains constrained by context and history trade-offs. The results vary across temporal scopes and capabilities, and the experimental protocol uses chunked streaming with a relaxed late-response boundary.
4.1 EXPERIMENTAL SETTING
- The streaming protocol sends one-second chunks, prompts non-streaming models to answer “Silent” when they need more context, and allows responses up to four seconds after an event ends.The context ceiling is 248,832,000 pixels, equivalent to 120 1080p frames; ProactiveFrame uses 90 focus frames and reserves at least 30 frames for sparse history.Source: §4.1 · pp. 7–8
4.2 MAIN RESULTS
- 50.7% is Gemini-3.5-Flash’s top overall accuracy; ProactiveFrame raises Qwen3-VL (8B) to 38.3% (+5.4%) and Gemini-3.5-Flash to 52.2%.JoyAI-VL-Interaction (8B) leads the open-source category at 44.1%.Source: §4.2 · p. 8
- With Qwen3-VL (8B), ProactiveFrame raises overall performance to 38.3% (+5.4%), including +37.5% in “Streaming & Online” and +8.6% in “Predictive & Causal” reasoning.Gemini-3.5-Flash declines in “Motion & Spatiotemp.” (-6.8%) and “Predictive & Causal” (- 5.9%); Qwen3-VL decreases in the “Instant” temporal scope (-2.2%).Source: §4.2 · p. 8
4.3 ABLATION ANALYSIS
- Full 30 FPS exceeds Mixed FPS on Forward queries for both models, while Backward queries peak under the oracle for both models.Qwen3-VL (8B) peaks under ProactiveFrame at 43.0%; Gemini-3.5-Flash peaks under Full 30 FPS.Source: §4.3 · pp. 8–9, Table 2 · p. 9
- Table 2 compares native 30 FPS, oracle Mixed FPS (30 FPS in human-annotated focus intervals and 2 FPS otherwise), and autonomous ProactiveFrame; the oracle scores 40.7% and 57.5%.ProactiveFrame scores 38.3% for Qwen3-VL (8B) and 52.2% for Gemini-3.5-Flash, while the oracle scores 40.7% and 57.5%, respectively.Source: §4.3 · p. 8, Table 2 · p. 9
- Qwen3-VL (8B) improves from 32.9% at 2 FPS to 44.6% at 24 FPS, then falls to 42.1% at 30 FPS as history compresses and frame redundancy rises.The drop at 30 FPS is most pronounced for Backward queries, which fall from 46.2% at 24 FPS to 37.5%; Instant queries remain strong at 30 FPS (46.3%).Source: §4.3 · p. 9, Table 3 · p. 9
5 CONCLUSION
FastBench evaluates high-dynamic perception in real-world streams, using trajectory-grounded verification and human review to construct its benchmark. Experiments show that denser sampling helps but is bounded by the granularity–history trade-off, while deciding when to look closer remains an open challenge.
5 CONCLUSION
- FastBench uses a trajectory-grounded pipeline to filter pseudo-dynamic questions, verify answers against SAM3 and CoTracker3 trajectories, and conduct multi-round human inspection.Gemini-3.1-Pro proposed questions and candidate intervals; human experts made the final decision for every released sample.Source: §5 · p. 9, AI USE STATEMENT · p. 10
- The strongest model reaches only 50.7%, and denser sampling helps within the granularity–history trade-off.ProactiveFrame improves over sparse sampling but trails oracle-guided focusing, leaving autonomous decisions about when to look closer as the central challenge.Source: §5 · p. 10
- The authors used generative AI to assist with question-answer annotation and answer judging, while human experts reviewed and finalized every released sample.They state that generative AI was not used to develop theoretical models, formulate or prove mathematical claims, propose hypotheses, design the method, or translate text.Source: AI USE STATEMENT · p. 10
A.1 MORE STATISTICS OF FASTBENCH
FastBench comprises 306 QA pairs across 300 clips, with distributions reported for clip duration and query timestamp. Queries occur 19.34 seconds after clip start on average.
A.1 MORE STATISTICS OF FASTBENCH
- FastBench contains 306 QA pairs across 300 clips; 294 clips have one pair each and six have two.The clips total 3.77 hours, averaging 45.29 seconds, with a 7.32–79.68-second range.Source: §A.1 · p. 13
- Queries occur 19.34 seconds after clip start on average, ranging from 0 to 74 seconds.The duration and query-timestamp distributions characterize the temporal context in which questions are presented.Source: §A.1 · p. 13
- Figure A.1 bins clip durations and query timestamps in 10-second intervals, with bar labels for counts and dashed lines for means.The query timestamps are measured from the start of each clip.Source: Figure A.1 · p. 13
A.2 DETAILS OF HUMAN ANNOTATORS
Ten annotators with multimodal video-understanding experience conducted the human verification, and each held at least a master’s degree.
A.2 DETAILS OF HUMAN ANNOTATORS
- Ten experienced multimodal video-understanding annotators conducted human verification, each holding at least a master’s degree and receiving $20 per hour.Source: §A.2 · p. 13
A.3 MORE DETAILED EVALUATION RESULTS
The appendix reports further FastBench evaluations across domains and capability–temporal-scope combinations using the protocol and scoring scale of Table 1.
A.3 MORE DETAILED EVALUATION RESULTS
- The additional evaluations cover the eight FastBench domains and capability–temporal-scope combinations.They use the same protocol and scoring scale as Table 1.Source: §A.3 · p. 13
- Domain scores average constituent QA pairs, while Overall averages all 306 pairs; domains are ordered by sample count.The domain results use the same protocol and scoring scale as Table 1.Source: §A.3 · p. 13
- Capability-wise results are further divided into Forward, Instant, and Backward subsets; all eight Streaming & Online Detection samples are Forward.That capability has no Instant or Backward samples.Source: §A.3 · p. 13
A.4 QUALITATIVE DATASET EXAMPLES
Four qualitative FastBench examples illustrate questions whose answers depend on brief visual changes.
A.4 QUALITATIVE DATASET EXAMPLES
- Each example shows three frames from the evidence interval alongside the query time, focus timestamp, and annotated answer.Each row is one QA pair, with the focus marking the timestamp requiring high-frame-rate perception.Source: §A.4 · p. 13
A.5 METADATA FILTERING
The appendix describes a metadata-based filter for locating high-dynamic video moments and ProactiveFrame instructions for adjusting frame rate during streaming inference.
A.5 METADATA FILTERING
- Table A.1 presents evaluation results across domains on a 0–100 scale; header parentheses give the number of QA pairs in each domain, and best displayed scores in each column are bolded.Source: Table A.1 · p. 15
- Timestamped comments and time-aligned danmaku are used as cues for high-dynamic moments, which Gemini-3.5-Flash identifies and timestamps to prune videos for later verification.The filter follows keyword retrieval; pure visual processing across hours of raw video is described as computationally prohibitive.Source: §A.5 · pp. 13, 15
A.6 PROACTIVEFRAME INSTRUCTION
- During streaming inference, Focus Start raises the next chunk’s frame rate, Focus End restores normal sampling, and Silent keeps the current mode while the model waits to answer.The instruction requires each control token to be the entire response.Source: §A.6 · p. 15, Figure A.4 · p. 18
- Table A.2 presents capability-wise evaluation results by temporal scope on a 0–100 scale; Fwd., Inst., and Back. mean Forward, Instant, and Backward, dashes mark empty subsets, and best displayed column scores are bolded.Source: Table A.2 · p. 16
- ProactiveFrame is instructed to monitor for potential brief or fine-grained events even without an active question, starting focus at a low threshold and ending it after the scene settles.High-frame-rate sampling begins with the next chunk after Focus Start; normal sampling resumes with the next chunk after Focus End.Source: §A.6 · p. 18