AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

1 Introduction

• Existing proactive benchmarks require waiting for sufficient evidence or adapt response timing, but generally assume the answer information will eventually appear in the stream.When information is incomplete, models may wait indefinitely or produce hallucinatory responses.Source: §1 · p. 2
• AskStream evaluates when and what to ask and when to answer across eight subtasks spanning video and user uncertainty, with manually annotated action windows, Ask-F1, and StreamHarness.It supports both multiple-choice and open-ended evaluation.Source: §1 · pp. 2–3
• Correct evidence lowers normalized answer entropy and shifts predictions toward the correct answer, unlike mismatched evidence.The reported accuracy rises from 42.3% to 62.3% after asking for key evidence.Source: §1 · p. 2, Figure 2 · p. 3
• The paper motivates a three-action proactive paradigm—wait, ask, and answer—by showing that acquiring missing information reduces model uncertainty.Source: §1 · pp. 2–3
• 52.18% and 7.22% Ask-F1 are achieved by the best offline and streaming proactive models, respectively.The experiments identify unreliable ASK decisions as a major bottleneck.Source: §1 · p. 3

2.1 Offline Paradigm

• Offline video benchmarks evaluate perception, temporal reasoning, and long-horizon understanding after models observe the complete video.Examples include Video-MME (Fu et al., 2025a), LVBench (Wang et al., 2025a), and LongVideoBench (Wu et al., 2024).Source: §2.1 · p. 3

2.2 Streaming Paradigm

• Streaming video benchmarks evaluate reasoning over content observed so far, including dynamic temporal reasoning and long-horizon interaction.Related methods target computational efficiency, GPU memory usage, and long-horizon memory.Source: §2.2 · p. 3

2.3 Proactive Paradigm

• Proactive benchmarks expand evaluation to when to answer and response timing, while information-asking work mainly targets static images and does not distinguish uncertainty sources.AskStream addresses this gap by modeling video and user uncertainty in streaming video.Source: §2.3 · pp. 3–4

3.1 Task Definition

• AskStream models streaming interaction as a POMDP with WAIT, ASK, or ANSWER decisions based on observed video, the query, and prior interactions.Type I events wait until evidence emerges before asking; Type II events ask first and then wait for answer evidence.Source: §3.1 · pp. 4–5

3.2 ASK-F1

• Ask-F1 jointly evaluates the timing and content correctness of both ASK and ANSWER actions.Its timing-aware asking and answering scores are denoted Sask and Sans, respectively.Source: §3.2 · pp. 5–6

3.3 Task Taxonomy

• The taxonomy divides uncertainty into video and user uncertainty, organized across four tasks and eight subtasks.Figure 3 presents examples of wait, ask, and answer interactions across the subtasks.Source: Figure 3 · p. 5, §3.3 · p. 6

3.3.1 Video Uncertainty

• Video uncertainty covers missing visual evidence and absent personalized concepts, including blurred entities or viewpoints and missing individual or action concepts.Source: §3.3.1 · p. 6

3.3.2 User Uncertainty

• User uncertainty includes ambiguous or false-premise questions and missing or outdated preferences, calling for clarification, correction, or updated preference input.Source: §3.3.2 · p. 6

3.3.3 Interaction Timing

• Explicit video uncertainty is recognized after observing relevant evidence, while implicit video and preference uncertainty can prompt asking from the outset.Intent uncertainty can follow either timing pattern, depending on when its ambiguity becomes recognizable.Source: §3.3.3 · p. 7

3.4 Benchmark Construction

• The five-stage pipeline collects or synthesizes videos, annotates queries and frame-level windows, modifies inputs, generates options, and manually verifies instances.Verification checks that answers require the missing evidence or clarification and that each candidate set has exactly one correct option.Source: Figure 3 · p. 5, §3.4 · p. 7

3.5 Benchmark Statistics

• 438 videos and 1,401 questions span eight subtasks and nine domains; 2,714 timestamps average 3.2 turns per video, with wait, ask, and answer windows totaling 33.92, 30.47, and 12.00 hours.Source: §3.5 · p. 7, Figure 4 · p. 7

4.1.1 Baseline

• The evaluation covers offline closed- and open-source models and streaming models with proactive capabilities, excluding streaming models without those capabilities.Source: §4.1.1 · p. 8

4.1.2 Evaluation protocols

• Models are evaluated at one action opportunity per second with ±1-second tolerance in multiple-choice and open-ended settings; StreamHarness retains recent frames and hierarchically summarizes older video context.Across turns, shared video memory is preserved, while questions, options, and action histories are discarded.Source: §4.1.2 · p. 8

4.2 Main Results

• 52.18% overall Ask-F1 for Gemini-3.8 Flash contrasts with AURA’s 7.22%, while streaming proactive models score zero on most subtasks or fail to complete the task.Offline models’ main error is asking too early; unreliable ASK decisions remain a key constraint.Source: §4.2 · pp. 8–9, Figure 5 · p. 8, Table 2 · p. 9

4.3.1 Where Do Models Fail?

• Providing oracle information about when to ASK raises Ask-F1 from 37.17% to 58.30% (+21.13 points), the largest improvement among the tested decision stages.Performance generally improves as more decision-stage uncertainty is removed.Source: §4.3.1 · p. 9, Figure 6 · p. 9

4.3.2 The Impact of Contextual Memory

• Contextual memory has little effect on video uncertainty but improves user-uncertainty performance by 4.40 points on AUQ and 5.38 points on UPM.The analysis attributes the gains to user intent and preferences persisting across turns.Source: §4.3.2 · p. 10, Figure 7 · p. 10

4.3.3 The Effect of Ask Budget

• ASK budgets from 1 to 16 improve performance, which gradually saturates at around 52.35%.The reported gains from additional ASK opportunities are limited.Source: §4.3.3 · p. 10, Figure 7 · p. 10

• AskStream evaluates when and what models should ask before answering across eight subtasks covering video and user uncertainty, using frame-level action windows, Ask-F1, and StreamHarness.Source: §5 · p. 10

• The benchmark permits at most one ASK per turn, excludes embodied agents coordinating perception and physical actions, assumes an explicit user query, and uses frame-level windows despite potentially longer, more complex uncertainty.The authors identify repeated clarification, autonomous detection of information insufficiency, and overlapping or changing ASK/ANSWER opportunities as future extensions.Source: §A.1 · p. 18

Ask-F1 aggregation

• Ask-F1 uses a harmonic mean of Sask and Sans, penalizing imbalance between information acquisition and subsequent answering more than an arithmetic mean.The harmonic aggregation requires both stages to perform well for a high overall score.Source: §A.2 · p. 18

Conditional answer score

• Sans is measured only on turns with a correct ASK, separating answer reasoning failures from failures to acquire the needed information.When Nask = 0, Sans and consequently Ask-F1 are defined as 0.Source: §A.2 · p. 18

Temporal weighting

• Linear temporal weighting gives maximum credit at the start of a valid action window and decreases smoothly toward its end.The authors choose it as bounded, parameter-free, and easy to interpret.Source: §A.2 · p. 19

Alternative temporal weighting functions

• The logarithmic and exponential variants alter within-window timing credit: logarithmic weighting gives relatively more credit to later valid actions, while exponential weighting favors earlier actions more strongly.The exponential variant uses a fixed decay coefficient of 2 as a stress test rather than tuning it to individual models.Source: §A.2 · p. 19

Temporal weighting sensitivity

• Model rankings remain unchanged under linear, logarithmic, and exponential weighting, although absolute Ask-F1 scores vary.Table 3 compares the seven API-evaluated models in the multiple-choice setting.Source: §A.2 · p. 19, Table 3 · p. 20

Multiple-ASK protocol

• The multiple-ASK protocol tests repeated information-seeking opportunities with budgets of 1, 2, 4, 8, or 16 actions while leaving the answer criterion unchanged.Invalid or redundant asks consume budget but do not immediately end the turn.Source: §A.2 · p. 20

A.3 Uncertainty Analysis

• The analysis uses ten chronological frames sampled at 1 Hz from causal history, with six checkpoints before asking and six after the evidence reveal.The correct evidence is revealed one second after asking; a mismatched condition reveals an irrelevant entity from another event in the same video.Source: §A.3 · p. 20
• Answer uncertainty is measured with normalized entropy after mean token log-probability scores for five answer texts are normalized into an answer distribution.The scores are computed from the ten-frame input and the answer tokens.Source: §A.3 · p. 20
• After the correct evidence appears, normalized answer entropy consistently decreases across three turns, while an irrelevant reveal leaves uncertainty near its previous level.The comparison associates the reduction with relevant evidence rather than additional observations or elapsed time alone.Source: §A.3 · p. 21
• As evidence accumulates, support for the gold entity raises its answer probability and lowers the probabilities of the four incorrect alternatives; irrelevant evidence does not produce this separation.Relevant evidence specifically shifts the model’s belief toward the correct answer.Source: §A.3 · p. 21
• After asking for key evidence, accuracy rises from 42.3% to 62.3% and mean gold-answer probability from 25.8% to 36.8%, approaching full visibility (63.1%; 36.6%).With blurred input, accuracy is 42.3% and mean gold-answer probability is 25.8%; full visibility achieves 63.1% accuracy and 36.6% mean gold-answer probability.Source: §A.3 · p. 21

A.4 Open-Ended Evaluation Results

• Gemini-3.8 Flash achieves the strongest overall open-ended performance, with 29.19% Ask-F1.Overall open-ended performance drops substantially compared with the multiple-choice setting.Source: §A.4 · p. 21
• Gemini-3.8 Flash has Sask of 23.34% versus Sans of 38.97%, indicating that proactive information acquisition remains a major bottleneck.Source: §A.4 · p. 21
• Streaming proactive models struggle especially severely: most score zero on nearly all subtasks, while AURA reaches only 4.01% Ask-F1.Source: §A.4 · p. 21
• Generating appropriate ASK and ANSWER responses is considerably more challenging than selecting among predefined candidates.Source: §A.4 · p. 21

A.5 Validation of the Open-Ended Judge

• Two human annotators independently judged ASK and ANSWER correctness on 300 sampled turns, with disagreements resolved by an additional annotator.Source: §A.5 · p. 21
• Gemini-3.8 Flash shows high agreement with human judgments for both ASK and ANSWER responses.Source: §A.5 · p. 21
• Agreement on Gemini-generated responses is comparable to agreement on non-Gemini responses, providing no clear evidence of self-preference in the sampled data.Source: §A.5 · p. 21

A.6 StreamHarness Ablation

• Removing hierarchical memory reduces Gemini-3.8 Flash’s multiple-choice Ask-F1 from 52.18% to 41.70%.Source: §A.6 · p. 22, Table 5 · p. 22
• Uniform sampling reaches 26.72% Ask-F1 with 128 frames and 16.99% with 64 frames, below StreamHarness’s 52.18%.Each valid temporal window contains at least one sampled frame, but its boundaries are not provided to the model.Source: §A.6 · p. 22, Table 5 · p. 22
• The sliding-window baseline keeps only the most recent 10 frames and removes mid- and long-term memory.Source: §A.6 · p. 22
• The results support preserving dense recent observations together with compressed historical context for proactive ASK and ANSWER decisions in streaming video.Source: §A.6 · p. 22

A.7.1 Annotation and Verification Interface.

• The annotation and verification interfaces support temporal action-window labeling and checks that masked inputs are insufficient while revealed evidence resolves the query.The general prompt allows WAIT, ASK, or ANSWER, with ASK only after confirming essential information is missing and ANSWER after the required ASK and trigger condition.Source: §A.7.1 · pp. 22–23, Figure 8 · p. 24, Figure 9 · p. 25

A.7.2 Task-specific Construction Details

• KVEM, VIM, and ICM construct missing-evidence scenarios requiring an entity reveal, a sufficient alternate viewpoint, or definitions for every ambiguous personalized alias.Their task-specific prompts direct models to request the relevant visual evidence or definitions, then wait for the answer trigger; verification checks that the initial context is insufficient and the requested evidence resolves the gap.Source: §A.7.2 · pp. 23–24, KVEM PROMPT CHOICE · p. 29, VIM PROMPT CHOICE · p. 29
• PACM uses synthesized novel personalized actions and hard negatives, with ASK candidates distinguishing the target action from alternatives.A correct ASK requests a uniquely discriminative action characteristic; after it, a textureless white-humanoid demonstration supplies the missing concept without appearance cues, and the prompt directs the model to wait for the trigger before answering.Source: §A.7.2 · pp. 25–26, PACM PROMPT CHOICE · pp. 29–30
• AUQ models request clarification for ambiguous queries, while WUQ models identify and correct false premises that become detectable from video evidence.AUQ returns user clarification; WUQ returns confirmation of the correction. Their prompts require a specific clarification or visually supported correction, with waiting before relevant evidence appears.Source: §A.7.2 · pp. 26–27, AUQ PROMPT CHOICE · p. 30, WUQ PROMPT CHOICE · p. 30
• UPM requests missing user preferences, whereas UPC requests an update when historical preferences may no longer apply to current circumstances.Correct preference requests uniquely distinguish the target entity; incorrect requests leave multiple candidates possible, and the prompts direct the model to ask for a concrete preference dimension rather than a generic restatement.Source: §A.7.2 · p. 27, UPM PROMPT CHOICE · p. 30, UPC PROMPT CHOICE · pp. 30–31

A.7.3 Temporal Window Annotation Consistency.

• ASK windows achieved 0.962 average tIoU (0.21 s start and 0.27 s end differences), while ANSWER windows achieved 0.971 (0.18 s start and 0.23 s end differences).Two annotators independently re-annotated 160 sampled instances, with 20 from each of the eight subtasks.Source: §A.7.3 · p. 27

VIM PROMPT OPEN / VIM ASK JUDGE PROMPT

• VIM asks for a concrete viewpoint that covers the occluded area, and its judge accepts only a useful view that would let the current question be answered.The prompt specifies viewpoint position, angle, and spatial coverage; it rejects merely requesting the entity named in the question.Source: VIM PROMPT OPEN · pp. 31–32, VIM ASK JUDGE PROMPT · p. 35

ICM PROMPT OPEN / PACM PROMPT OPEN / ICM ASK JUDGE PROMPT / PACM ASK JUDGE PROMPT

• ICM asks must name every private entity alias, while PACM asks must name every action; explicit definition-request wording is optional for both.Their judges reject omitted or incorrect names and unrelated additions.Source: ICM PROMPT OPEN · p. 32, ICM ASK JUDGE PROMPT · p. 36, PACM ASK JUDGE PROMPT · p. 36

AUQ PROMPT OPEN / AUQ ASK JUDGE PROMPT

• AUQ asks must identify the specific ambiguity and seek clarification that resolves the request to one unique referent, rather than make a generic information request.The prompt also requires exact reproduction of any specified reminder message, including capitalization and punctuation.Source: AUQ PROMPT OPEN · p. 33, AUQ ASK JUDGE PROMPT · p. 36

WUQ PROMPT OPEN / WUQ ASK JUDGE PROMPT

• WUQ asks must identify the specific video-question contradiction and state a visually supported correction or ask the user to confirm it.Before evidence of the mismatch appears, the prompt directs the model to wait; afterward, it answers when the corrected trigger condition is satisfied.Source: WUQ PROMPT OPEN · p. 33, WUQ ASK JUDGE PROMPT · p. 37

UPM PROMPT OPEN

• UPM asks for the concrete preference dimension needed to answer and selects a feature whose values uniquely distinguish the relevant entity.The prompt rules out generic preference questions and features shared by multiple entities.Source: UPM PROMPT OPEN · p. 34

UPC PROMPT OPEN

• UPC asks the user to confirm or update the specific outdated or conflicting preference dimension, rather than restate all preferences.It likewise selects a dimension whose possible values uniquely distinguish an entity.Source: UPC PROMPT OPEN · p. 34

KVEM ASK JUDGE PROMPT

• KVEM responses must request the intended occluded entity and locate it by position or visible spatial context, not merely name its category.The judge rejects broad scene descriptions, unrelated targets, and missing or conflicting location information.Source: KVEM ASK JUDGE PROMPT · p. 35

WUQ ASK JUDGE PROMPT

• The WUQ ASK judge is instructed to evaluate only the supplied labeled text and images and return a single-digit response.The excerpt specifies this output constraint for an incorrect candidate.Source: WUQ ASK JUDGE PROMPT · p. 37

UPM ASK JUDGE PROMPT

• The UPM judge accepts a question about one concrete, visually grounded preference dimension only if the Correct Image’s value uniquely identifies it.The requested feature must be comparable across option images, and the Correct Image’s value cannot be shared by another image.Source: UPM ASK JUDGE PROMPT · p. 37

UPC ASK JUDGE PROMPT

• The UPC judge accepts a question only when it elicits one concrete, visually comparable preference value that uniquely identifies the Correct Image.It rejects questions that fail to request a current or updated preference, combine dimensions, or conflict with the Correct Image’s value.Source: UPC ASK JUDGE PROMPT · pp. 37–38

A.9 Study Case

• Study cases illustrate video questions answered after clarification, questions grounded in scene details, and outfit selection using stated or updated preferences.Examples include a user reply about a woman’s striped top, identifying a wall color, selecting an outfit from a stated preference, and choosing a sofa using an updated preference.Source: §A.9 · pp. 40–42