Source-linked AI summary
Foresight: planning future perception in streaming VLMs without retraining
Ashok Prasad Neupane, Dipan Bartaula, Ankit Belbase, Saugat Adhikari, Samip Ghimire, Saroj Poudel, Binod Bhattarai, Danda Pani Paudel
TL;DR
Streaming VLMs use fixed computation even as scenes evolve, leaving a gap in how they adapt what and when they perceive. FORESIGHT plans future computation asynchronously with a frozen VLM, reaching 23.0 mean F1 versus 13.5 for the strongest baseline on OmniPro Online and improving streaming-understanding benchmarks.
Problem
Existing streaming VLMs only partly address response timing, computation cost, and asynchronous reasoning, and still make computation decisions reactively from arrived evidence rather than anticipated events.
Source: §1 · p. 2Method
Results
23.0 mean F1 versus 13.5 for MiniCPM-o 4.5 on OmniPro Online; FORESIGHT also improves StreamingBench by 6.7 points and OVO-Bench by 15.4 points.
Source: §1 · p. 2Takeaways & Limitations
The results support anticipation as a way to control computation, with the largest reported gain where evidence arrives later in the video stream.
Source: §1 · pp. 2–3Reliable operation depends on the backbone's future anticipation and temporal grounding; false-trigger reporting can persist, and control remains sensitive to prompt wording.
Source: §6 · pp. 9–10
Abstract
from arXiv · showhide
Existing streaming vision-language models (VLMs) continuously perceive and reason over visual streams, but their computational pathways remain fixed throughout inference. Consequently, they cannot adapt computation to evolving scene dynamics, where different future events demand different levels and forms of perception. We show that streaming VLMs inherently possess the ability to anticipate the immediate future, and leverage this capability to dynamically configure future computation in a training-free manner. Realizing such anticipatory computation, however, is very challenging: future anticipation must be sufficiently reliable to guide computation, planning must run concurrently with streaming inference, and online reconfiguration must incur negligible overhead. To address these challenges, we introduce FORESIGHT, a dual-stream architecture comprising two Siamese LLMs with shared weights, input encoders, and KV cache. The first LLM continuously processes incoming tokens, while the second runs ahead of the stream to anticipate future context, plan future computation, and generate task responses without interrupting streaming inference. Each plan decides when to reason next, what to check then, and how densely to sample, keeping transient evidence separate from persistent control. The resulting computation plan is executed online through an efficient reconfiguration protocol with schemaguided decoding and lightweight diff-based updates, enabling dynamic adaptation with low overhead. With a frozen Qwen3-VL-8B backbone, FORESIGHT achieves 23.0 mean joint F1 on OmniPro Online evaluation beating strongest trained baseline by 9.5%, while improving the backbone by 6.7 on StreamingBench and 15.4 on OVO-Bench, with the largest gain of 18.7 when evidence arrives later in the video stream. Our source code will be made publicly available.
1 INTRODUCTION
Existing streaming VLMs make fixed or reactive computation decisions, while evolving scenes require deciding when to check, what to inspect, and how much computation to use. FORESIGHT uses a frozen VLM to plan those decisions asynchronously, without retraining or pausing perception.
1 INTRODUCTION
- Existing streaming methods may time responses, reduce per-step cost, compress memory, or reason asynchronously, but still base computation on arrived evidence rather than anticipated events.Those systems leave the wait for evidence to a fixed schedule or a learned trigger.Source: §1 · p. 2
- FORESIGHT treats anticipation as a way to control computation, letting the model decide what remains uncertain, when to check again, and how densely to sample.A second copy plans in the background while a frozen VLM continues ingesting the stream, so planning requires no retraining and does not pause perception.Source: §1 · p. 2
- FORESIGHT addresses reliability, concurrency, and efficiency by separating transient evidence from persistent controls, using Siamese Ingest and Think copies, and decoding only plan values that require reasoning.Hysteresis on evidence confidence gates responses to reduce premature or repeated outputs.Source: §1 · p. 2
- 23.0 mean F1 versus 13.5 for MiniCPM-o 4.5 on OmniPro Online; FORESIGHT also improves StreamingBench by 6.7 points and OVO-Bench by 15.4 points.Its largest gain, 18.7 points, is on Forward Active Responding, where the answer appears later in the video.Source: §1 · p. 2
2 RELATED WORK
Prior streaming VLM work addresses response timing, proactive behavior, asynchronous reasoning, and per-step cost reduction. FORESIGHT's distinction is to use anticipation to plan future computation while the stream continues.
Streaming and proactive vision-language models
- Streaming systems progressed from learned silence-or-response decisions to proactive and always-on interaction, while methods such as LiveStar, Dispider, StreamReady, and QueryStream use different readiness or triggering signals.Several of these methods rely on training or repeated decoding to decide whether to respond.Source: §2 · p. 3
Asynchronous Computation
- Asynchronous systems separate monitoring from heavier reasoning or keep reading input while thinking, but generally react to present input; FORESIGHT instead uses asynchronous twin VLMs with a shared KV cache.StreamAgent feeds predictions back into computation, but uses a fixed planning schedule and candidate plans scored by an LLM judge.Source: §2 · p. 3
Anticipatory Computation
- FORESIGHT's planning role differs from action anticipation as a final prediction: it uses a plan to choose when the next reasoning step occurs, what question it resolves, and how densely to sample meanwhile.The overview depicts a Think LLM reading a cache snapshot and writing an evidence state and control update for the Live Executor.Source: §2 · p. 3, Figure 2 · p. 4
Pruning and Self-programming
- Pruning methods reduce cost by trimming frames or tokens that have already arrived, whereas FORESIGHT plans the sampling density of future frames and task-informed token pruning.The related-work discussion also distinguishes model-driven reconfiguration from systems that rewrite procedures over fixed input.Source: §2 · pp. 3–4
3 FORESIGHT
FORESIGHT equips an off-the-shelf frozen VLM for proactive streaming by planning computation as activity evolves. Its asynchronous architecture separates ongoing perception from planning, and its dynamic loop adjusts sampling, attended objects, and cache state over time.
3 FORESIGHT
- FORESIGHT is a training-free method that equips an off-the-shelf VLM for proactive streaming tasks by dynamically reconfiguring computation according to evolving stream activity.It combines asynchronous perception, planning, and response generation for real-time human–computer interaction.Source: §3 · p. 4
3.2 DYNAMIC BEHAVIOUR
- At scheduled checks, FORESIGHT applies the previous plan's controls and sets the next check; in the goal example, sampling rises from 1 fps to 8 fps as a shot develops, then returns to 1 fps after the answer.The plan can add the goalkeeper and net, compact the cache, and later drop those classes once evidence is sufficient.Source: Figure 3 · p. 5, §3.2 · p. 6
3.1 ARCHITECTURAL OVERVIEW
- A continuous Ingest LLM writes admitted-frame tokens to a shared KV cache, while a Think LLM reads a snapshot and plans without blocking ingestion or writing into the visual history.The cache communicates accumulated context; it is bounded by a sliding window or compacted when requested, while the Vision Gate and Encoder adjust frame sampling and token pruning.Source: §3.1 · pp. 4–5
4 EXPERIMENTS
The experiments evaluate a frozen Qwen3-VL-8B on proactive responding and streaming-understanding benchmarks, alongside controller and input-processing ablations. FORESIGHT is best on the audio-free OmniPro Online subset, and the experiments describe its implementation and evaluation scope.
4.1 IMPLEMENTATION DETAILS
- FORESIGHT uses a frozen Qwen3-VL-8B backbone and feeds consecutive frames as two-frame video clips through Qwen3-VL’s native video pathway.The input-processing ablation compares these clips with individually encoded frames.Source: §4.1 · pp. 6–7
- Plan updates rewrite changed fields while retaining omitted settings; evidence and response fields are recomputed from the current stream, and the live executor applies each new plan.Boolean decisions use relative yes/no token probabilities as probes.Source: §4.1 · p. 7
- The controller can prune visual tokens using kept or ignored classes, while plan-selected frame rates control sampling density.The plan’s fps field sets the visual sampling rate.Source: §4.1 · p. 7
- Per-task response thresholds and gate settings are fitted on a random 20% calibration split; FORESIGHT results use the held-out 80%.Baseline values are reported on full evaluation sets.Source: §4.1 · p. 7
4.2 BENCHMARKS
- 43.7 mean timing F1 and 54.6 content accuracy on on-time responses; FORESIGHT performs best on audio-free OmniPro Online joint F1.The benchmark evaluates nine task types under the official protocol.Source: §4.2 · p. 7, Table 1 · p. 8
- StreamingBench covers real-time visual understanding, omni-source, and contextual tasks; OVO-Bench groups tasks by whether evidence is current, earlier, or still to come.Tables 2 and 3 summarize results for the respective benchmarks.Source: §4.2 · pp. 7–8
5 CONCLUSION
FORESIGHT applies anticipation to decide how a frozen VLM spends computation during streaming. Its results support the approach across proactive responding and streaming understanding, with the largest reported gain when evidence appears later in the stream.
5 CONCLUSION
- 23.0 joint F1 on OmniPro Online versus 13.5 for the strongest trained baseline, while running close to real time.FORESIGHT also achieves the best overall score on StreamingBench and improves its backbone by 15.4 points on OVO-Bench.Source: §5 · p. 9
- 18.7 points is FORESIGHT’s largest reported gain, on Forward Active Responding when the answer appears later in the stream.The same frozen-backbone controller can directly benefit from stronger backbones without updating weights.Source: §5 · p. 9
- Two copies of the same model share a KV cache: one ingests the stream while the other plans when to reason, what to check, and how densely to sample.The paper frames anticipation as a way to allocate computation to what comes next, rather than as a prediction to score.Source: §5 · p. 9
6 LIMITATIONS AND FUTURE WORK
FORESIGHT’s reliability depends on the underlying VLM’s ability to anticipate future events and localize them in time. The paper also reports persistent false-trigger behavior and sensitivity to prompt wording.
6 LIMITATIONS AND FUTURE WORK
- Reliable operation depends on the backbone’s capacity for future anticipation and temporal event localization.The paper identifies stronger backbones or dedicated inference-time mechanisms as directions for improving these capabilities.Source: §6 · p. 9
- After a false trigger, the model can continue reporting an event without supporting observations; history reduces but does not eliminate this behavior.The controller also remains sensitive to prompt wording.Source: §6 · p. 10
7 AI USE STATEMENT
The statement describes the paper’s use of generative AI tools and identifies GPT-4o-mini as an automatic content-correctness judge for two benchmark tasks.
7 AI USE STATEMENT
- Generative AI assisted with parts of evaluation and inference code and with correcting LaTeX formatting.The authors state that AI tools were not used to develop the conceptual framework, hypotheses, methodology, experiments, or interpretation.Source: §7 · p. 10
- GPT-4o-mini serves as an automatic content-correctness judge for Event-Narration and Step-Instruction.The statement places this use within the benchmark protocol.Source: §7 · p. 10
- The authors report reviewing and testing AI-assisted code and taking responsibility for the paper’s content.Source: §7 · p. 10
B.1 COMPUTE INFRASTRUCTURE
The appendix reports the evaluation setup and explains how FORESIGHT uses in-context planning without changing model weights. Plans combine scene knowledge with task-specific timing, sampling, and follow-up checks.
B.1 COMPUTE INFRASTRUCTURE
- Full-benchmark experiments ran on NVIDIA GH200 GPUs, and the plan’s fps field sets visual sampling rate.Removing sampling-rate control is evaluated in Table 4.Source: §B.1 · p. 17
- Each benchmark’s evaluation set is split into 20% calibration for response thresholds and 80% held-out evaluation for reported FORESIGHT results.A full OmniPro configuration evaluation took about 150 GH200 GPU-hours.Source: §B.1 · p. 17
B.2 IN-CONTEXT LEARNING AND EVALUATION PROMPTS
- FORESIGHT uses frozen-model world knowledge and in-context learning to translate expected events into plans, without changing weights.Examples adjust check timing and fps for an anticipated goal or boiling water, and use reported events to avoid duplicate counts.Source: §B.2 · p. 17
- Evaluation uses each benchmark’s official task prompt, fixed as a persistent KV-cache prefix; planner text is not inserted into visual history.Prompts are not revised after seeing results, and response thresholds use only the calibration split.Source: §B.2 · p. 17
C.1 ONLINE-MODE TASK BREAKDOWN
The task breakdown distinguishes response timing, content correctness, and their joint score, while the baseline analysis shows that speaking frequency alone does not determine timing performance.
C.1 ONLINE-MODE TASK BREAKDOWN
- Dispider’s results cover a 932-example subset and are not directly comparable with full-benchmark rows; MiniCPM-o 4.5’s published mean Joint F1 is 13.8.The cited published per-task scores are for the same model and setting; StreamAgent’s macro means are 12.5 Time F1, 43.0 content, and 5.0 Joint F1.Source: §C.1 · p. 18
- Time F1, content accuracy, and Joint F1 decompose the same emissions; per task, Time F1 multiplied by content equals Joint F1.The Overall column may not multiply out because sources pool it differently.Source: Table 5 · p. 18
C.2 EMISSION BEHAVIOUR OF THE BASELINES
- MiniCPM-o 4.5 emits 0.37 responses per event versus StreamAgent’s 0.79, yet has higher Time F1, 24.4 versus 15.0; QueryStream emits 2.75 per event.QueryStream’s Time F1 is 17.7, but most responses are wrong, keeping Joint F1 low.Source: §C.2 · p. 18
D.1 TASK-LEVEL BREAKDOWN
The task-level analysis separates streaming accuracy tasks from the timing-focused Proactive Output task. Its reported gate score is treated as diagnostic given wide confidence intervals and weaker comparison to the strongest proprietary systems.
D.1 TASK-LEVEL BREAKDOWN
- StreamingBench Table 6 reports per-task Real-Time Visual Understanding accuracy (%); its first two blocks are reference points from Lin et al. (2024b), and plain frame counts indicate uniform sampling, not streaming rates.SQA denotes Sequential Question Answering, and PO denotes Proactive Output.Source: Table 6 · p. 19, §D.1 · p. 19
D.2 PROACTIVE OUTPUT
- Proactive Output scores whether a response arrives within the annotated timing tolerance, unlike the other seventeen StreamingBench tasks, which score answer content.It is excluded from the Contextual average; FORESIGHT’s average across all four tasks is 44.50.Source: §D.2 · p. 19, Table 7 · p. 20
- 46.3 (95% CI 39.1–53.7) is the official-protocol Proactive Output score, with median timing error of 4.0 seconds and silence on 15 of 175 questions.Seeing the full clip made the gate fire on every question but left the score unchanged.Source: §D.2 · p. 20
- Proactive Output is a diagnostic rather than a headline result because its few questions produce wide confidence intervals; the gate is indistinguishable from Gemini 1.5 Pro and behind Claude 3.5 Sonnet and GPT-4o.The cited scores are 45.1 for Gemini 1.5 Pro, 64.7 for Claude 3.5 Sonnet, and 56.9 for GPT-4o.Source: §D.2 · p. 20
E ADDITIONAL OVO-BENCH ANALYSIS
The additional OVO-Bench analysis clarifies benchmark versions, model configurations, and the scope of its task-level tables. It distinguishes real-time visual perception from temporal reasoning, including backward tracing and forward active responding.
E ADDITIONAL OVO-BENCH ANALYSIS
- The OVO-Bench analysis divides results into Real-Time Visual Perception and temporal-reasoning task groups, with human scores based on native-rate video viewing.The StreamBridge row uses its Qwen2-VL-7B plus Stream-IT configuration; its LLaVA-OV-7B backbone has separate reported scores.Source: E ADDITIONAL OVO-BENCH ANALYSIS · p. 21
- OVO-Bench v2 changed most baseline scores from v1, so the reference rows use v2 for consistency; method-paper rows retain their own reported values.TimeChat-Online configurations differ across OVO-Bench and StreamingBench because each comparison uses its reported keep rate.Source: E ADDITIONAL OVO-BENCH ANALYSIS · p. 21
E.1 REAL-TIME VISUAL PERCEPTION
- Table 8 reports per-task Real-Time Visual Perception accuracy, with reference models from the current OVO-Bench v2 release.Qwen3-VL-8B and MiniCPM-o 4.5 results are attributed to Lu et al. (2026).Source: Table 8 · p. 21
E.2 BACKWARD TRACING AND FORWARD ACTIVE RESPONDING
- Table 9 breaks down temporal-reasoning accuracy; VideoLLM-Online’s Forward Active Responding and overall values come from the authors’ run, not the original release.The task labels include Episodic Memory, Action Sequence Identification, and Forward Active Responding.Source: §E.2 · p. 21, Table 9 · p. 22
F ANTICIPATION ANALYSIS
Anticipation is most useful over short horizons, so FORESIGHT renews scene-dependent checks while keeping their compute overhead low. The appendix also reports runtime comparisons and the per-task response-gate settings used for OmniPro Online.
F ANTICIPATION ANALYSIS
- Two seconds before an event, pooled anticipation closes about three quarters of the blind-to-oracle gap; by 30 seconds, that share falls to about a tenth.Figure 4 shows pooled scores approaching the blind baseline as lead time grows.Source: F ANTICIPATION ANALYSIS · p. 23, Figure 4 · p. 23
- Because foresight is reliable only a few seconds ahead, FORESIGHT renews checks often, lengthening intervals in calm scenes and shortening them as activity develops.Current VLM foresight fades within seconds, so reducing checks to once per event or a few per minute on calm streams remains an open problem; the paper proposes training models to anticipate events, timing, or the next useful moment to look.Source: F ANTICIPATION ANALYSIS · pp. 23–24
G INFERENCE EFFICIENCY
- Shared caching, adaptive frame sampling and pruning, short probes, and diff-based updates reduce repeated ingest and planning work.The probe decodes only the plan's first field when no new evidence is found; when evidence arrives, the model writes only changed fields.Source: G INFERENCE EFFICIENCY · p. 25
- Controller cost grows with the number of checks and events, while a fixed cache budget keeps each decision's cost from increasing over time.Above the oracle, FORESIGHT's extra work is approximately K cprobe, with a small probe cost and a video-dependent check count.Source: G INFERENCE EFFICIENCY · p. 25
- At one check every four seconds, the cost model estimates about 40× higher total compute without shared caching and roughly 3× without adaptive sampling and pruning.The modeled overhead above the oracle is about 25× for full plans instead of diffs, 4× for diffs at every check without a probe, and 2× for a separate probe pass.Source: G INFERENCE EFFICIENCY · p. 26
- FORESIGHT's total compute is barely distinguishable from the oracle, and its extra compute is the cheapest among the designs considered.Figure 5 compares total compute; Figure 6 compares compute above the oracle.Source: Figure 5 · p. 26, G INFERENCE EFFICIENCY · p. 26, Figure 6 · p. 27
G.1 MEASURED COST OF THE STREAMING BASELINES
- On one GH200, StreamAgent's full arm costs 2.13× and 2.70× video duration on its two backbones, while its probe configuration costs 0.58×; QueryStream costs 0.053× and 0.084×.The measurements use the OmniPro Online set; both QueryStream backbones run within real time.Source: §G.1 · pp. 26–27
H CONTROLLER CONFIGURATION AND CODE RELEASE
- OmniPro Online gate settings were fitted per task on the 20% calibration split and held fixed on the 80% evaluation split.Edge gates with long refractory intervals suit one-off alerts and counts; level gates with short intervals suit changing-state tasks.Source: H CONTROLLER CONFIGURATION AND CODE… · p. 28, Table 10 · p. 28