StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos

1. Introduction

• Streaming video models process incoming frames without future context, a requirement for continuous perception and decision-making in applications such as robotics and AR-glass assistants.The introduction also mentions embodied agents as a target application.Source: §1 · p. 2
• Existing streaming benchmarks cover only some realistic challenges: some emphasize past or present recognition, while proactive understanding and human perceptual signals such as gaze are rarely included.The introduction identifies gaze as a cue to where users attend, what they process, and what they may do next.Source: §1 · p. 2
• Gaze-guided QA is challenging because noisy gaze trajectories must be grounded in a moving egocentric view, while streaming models can use only past and currently observed frames.The gaze shifts and camera motion make even identifying what the user is looking at a spatio-temporal grounding problem.Source: §1 · p. 2
• STREAMGAZE evaluates gaze-guided past, present, and proactive reasoning using only past and currently observed frames.Models must interpret gaze online, follow shifting attention, and infer intention.Source: §1 · p. 2
• Its semiautomatic pipeline aligns gaze trajectories with egocentric video, extracts fixations and fixated objects, constructs scanpaths, then generates and verifies streaming QA pairs.An LLM generates the tasks, followed by human verification.Source: §1 · p. 2
• Across STREAMGAZE tasks, state-of-the-art MLLMs show substantial performance gaps from both humans and specialized streaming video understanding models.The analyses also find difficulty leveraging gaze for temporal reasoning and proactive inference.Source: §1 · p. 2

Streaming QA benchmarks

• STREAMGAZE integrates eye-gaze dynamics into streaming evaluation of temporal reasoning, intention inference, and proactive prediction.The cited streaming benchmarks evaluate online video understanding and interactive dialogue, while the cited gaze-based benchmarks do not cover the online past-present-proactive setting.Source: §2 · p. 3

Gaze-based QA benchmarks

• GazeVQA addresses collaborative assembly interactions, while EgoGazeVQA focuses on intent in short egocentric videos using frame-wise gaze without fixation extraction or spatio-temporal QA grounding.Neither addresses online past, present, and proactive tasks.Source: §2 · p. 3

3. Gaze-Guided Streaming Data Construction

• The construction pipeline aligns full gaze scanpaths with egocentric video, extending beyond static per-frame gaze points to capture attention shifts and objects entering the FOV through ego-motion.Fixations identify meaningful attention events before region-based object extraction and human verification.Source: §3 · p. 3

3.1. Preprocessing

• The pipeline uses EGTEA+, EgoExoLearn, and HoloAssist videos, projecting world-coordinate gaze onto image frames with camera parameters when needed.Datasets with provided 2D gaze coordinates use them directly.Source: §3.1 · p. 3

3.2. Fixation Extraction

• Fixations require spatial concentration and sufficient duration, then pass a scene-consistency check that rejects discontinuous segments.The gaze dispersion threshold is normalized by frame width; scene consistency uses adjacent-frame hue–saturation histogram correlation.Source: §3.2 · p. 4

3.3. Object Extraction by Gaze Area

• Scanpaths preserve fixation order by linking FOV and out-of-FOV object sets, capturing attention shifts across spatial regions and semantic contexts.The resulting gaze-conditioned trajectory supports temporally grounded QA construction.Source: §3.3 · p. 5

3.4. Human Verification

• Human annotators correct object labels and captions in generated scanpaths; only verified data enter the benchmark, with approximately 83% average correctness.Verification covers whether fixation-level objects should be included or excluded.Source: §3.4 · p. 5

4. STREAMGAZE

• STREAMGAZE introduces a QA-generation process and taxonomy for quantitatively evaluating gaze-guided streaming video understanding across past, present, and proactive tasks.The benchmark contains over 8,521 QA pairs from 285 videos and covers 10 tasks.Source: §4 · p. 5, §4.1 · p. 5

4.1. Benchmark Overview

• The benchmark contains over 8,521 QA pairs from 285 videos, with 10 tasks grounded in gaze and egocentric video.At query time, models answer time-sensitive questions from incoming frames and gaze within a 60-second temporal window.Source: §4.1 · p. 5

4.2. Task Taxonomy

• The task taxonomy constructs QA pairs requiring spatio-temporal reasoning over gaze transitions.Questions use the accessible portion of the video and gaze stream to define past, present, and proactive tasks.Source: §4.2 · p. 5

4.2.1. Past Task

• Past tasks test gaze history through identifying never-fixated visible objects, predicting the next fixation, matching gaze sequences, and recalling background objects.Together, they target attention shifts, temporal continuity, scanpath patterns, and contextual memory.Source: §4.2.1 · pp. 5–6

4.2.2. Present Task

• Present tasks assess current perception and intention through attended-object identification, attribute recognition, and prediction of the user’s next action.Object identification varies distractor difficulty, while future actions use recent gaze-conditioned context and aligned human action captions.Source: §4.2.2 · p. 6

4.2.3. Proactive Task

• Proactive tasks use gaze as an anticipatory signal: Gaze-Triggered Alert marks fixation, while Object Appearance Alert reacts to a specified object entering the peripheral region.Ambiguous questions are filtered through model-based validation and human verification.Source: §4.2.3 · p. 6

5.1. Experimental Setup

• Models were evaluated zero-shot with visual gaze prompts; non-streaming models received offline video clips, and human-oracle performance was reported.Past and present tasks use accuracy, while proactive reminders use multi-triggering accuracy; the benchmark table reports proactive precision.Source: §5.1 · pp. 6–7, Table 2 · p. 7

5.2. Overall Performance

• General-purpose MLLMs struggle with long-term gaze reasoning, in part because sparse key-frame inputs impede accumulation of gaze evidence across a continuous stream.The passage attributes this limitation to constrained model capacity and processing frames independently rather than as an evolving stream.Source: §5.2 · p. 7
• ViSpeak outperforms open-source non-streaming MLLMs on proactive tasks, while dialogue-based VideoLLM-online often produces generic descriptions instead of following instructions.The passage attributes the proactive limitation to streaming architectures lacking explicit mechanisms linking gaze shifts to predictive temporal reasoning.Source: §5.2 · p. 7
• AssistGaze achieves only 0.223 average accuracy and struggles to generalize from static or short-range settings to proactive, long-horizon streaming tasks.Source: §5.2 · p. 7
• Humans significantly outperformed all MLLMs at 0.827 avg. accuracy, with disparities exposing weak scene-context and gaze-transition modeling.The reported Scene Recall and Object Transition Prediction disparity further highlights these weaknesses.Source: §5.2 · p. 7

5.3. Additional Analysis

• The salience-map prompt performs best, particularly on present and proactive tasks, but no gaze-prompting strategy consistently beats the gaze-free baseline.Table 3 reports the gaze-input prompting ablation for Qwen2.5-VL.Source: Table 3 · p. 8, §5.3 · p. 8
• GPT-4o achieves its best average performance when textual, gaze-based, and visual reasoning strategies are combined, though visual cues help Scene Recall and can hinder Non-Fixated Object Identification.The ablation compares chain-of-thought prompting, estimating gaze before answering, and identifying visible objects with bounding boxes.Source: §5.3 · p. 8, Table 4 · p. 8
• Task responses vary across gaze and reasoning strategies, and proactive error profiles differ by model.InternVL3.5-8B over-triggers, GPT4o errors rise on harder tasks, and Qwen2.5-VL-7B is more balanced; Figure 5 distinguishes false-positive and false-negative errors.Source: Figure 5 · p. 8, §5.3 · p. 8, Figure 4 · p. 8

• STREAMGAZE is introduced as a benchmark for evaluating gaze-guided temporal and proactive reasoning in streaming video understanding.Source: §6 · p. 8
• Its gaze–video QA pipeline provides human-aligned supervision across past, present, and proactive tasks.Source: §6 · p. 8
• Experiments reveal substantial gaps between state-of-the-art MLLMs and human annotators, especially in intention modeling and proactive prediction.The authors hope the benchmark advances human-aligned, attention-aware streaming models.Source: §6 · p. 8

A.1. Gaze Projection

• Gaze rays are projected into frame-level 2D positions using camera poses and intrinsics, with nearest-timestamp alignment for video, pose, and gaze signals.Nearest-neighbor alignment accommodates differing sampling rates, and the projected coordinates form the gaze trajectory.Source: §A.1 · p. 11, Algorithm 1 Gaze-Guided Data… · pp. 11–12

A.2. Fixation Extraction

• Fixation extraction tolerates gaze deviations lasting at most 200 ms as transient flicks and retains intervals only when scene-frame HSV histogram similarity meets an approximately 0.9 threshold.Intervals failing the scene-consistency check are discarded as including scene changes.Source: §A.2 · p. 12

A.3. Object Extraction by Gaze Area.

• Gaze-area object extraction uses a 15° perifoveal radius and region-specific prompts to identify gazed and nearby objects or discover objects outside a masked central region.When camera intrinsics are unavailable, the pipeline assumes a 90° HFOV; prompts enforce consistent object naming and grounded descriptions.Source: §A.3 · pp. 12–13

A.4.1. Non-Fixated Object Identification (NFI)

• NFI identifies visible but never-fixated objects, queries once three unique fixation objects have accumulated, and sets response windows around distractor visibility with two-second margins.Answers are drawn from visible, never-fixated objects, and human-verified scanpaths receive formatting and consistency checks.Source: §A.4.1 · p. 13

A.4.2. Scene Recall (SR)

• SR asks which background object was previously visible but is absent during the current fixation, querying when the user first fixates on an object named in the question.Qwen3-VL filtering checks question clarity, that the answer was never fixated, and that it appears in the video.Source: §A.4.2 · p. 13

A.4.3. Object Transition Prediction (OTP)

• OTP predicts the next newly attended object, querying at the onset of the current fixation group and spanning the transition with two-second margins.Transitions come from human-verified scanpaths, so no additional visual filtering is applied.Source: §A.4.3 · p. 13

A.4.4. Gaze Sequence Matching (GSM)

• GSM matches consecutive three-step fixation sequences against shuffled or random alternatives, querying at the third group’s end over the full sequence interval.Qwen3-VL removes samples when a distractor sequence overlaps the correct sequence in more than 50% of segments.Source: §A.4.4 · pp. 13–14

A.4.5. Object Identification (OI, Easy/Hard)

• OI Easy/Hard evaluates recognition of the currently fixated object within the gaze-aligned cropped region.Source: §A.4.5 · p. 14
• OI easy-task distractors come from elsewhere in the video, while hard-task distractors are visible in-frame but outside fixation; samples lacking enough distractors are discarded.The query timestamp is the start of the fixation.Source: §A.4.5 · p. 14

A.4.6. Object Attribute Recognition (OAR)

• OAR uses fixed attribute-specific templates, with Qwen3-VL-30B selecting an attribute, extracting a caption-grounded answer, and generating three plausible distractors verified for ambiguity [30].The task probes attributes such as color, material, shape, texture, size, or state of the fixated object.Source: §A.4.6 · p. 14

A.4.7. Future Action Prediction (FAP)

• FAP labels the earliest annotated action 3 seconds to 1 minute after a three-fixation sequence ends, using same-video actions as filtered distractors.The query is issued when the sequence ends, and the response window starts two seconds before the action to avoid temporal leakage.Source: §A.4.7 · p. 14

A.4.8. Gaze-Triggered Alert (GTA)

• GTA samples timestamps around each object’s first fixation and assigns binary fixation labels, filtering static structures with Qwen3 and manually verifying labels [30].The sampled offsets include 20 and 10 seconds before the event, the event time, and 10 and 20 seconds after it.Source: §A.4.8 · p. 14

A.4.9. Object Appearance Alert (OAA)

• OAA labels whether an object is visible outside fixation at timestamps around its first appearance, with Qwen3-VL confirmation and manual verification [30].The procedure filters background structures and checks visibility within a 10-second evaluation window.Source: §A.4.9 · p. 14

A.5. Human Verification

• Fleiss’ Kappa was 0.60; three annotators verified extracted objects and corrected gazed-object identities and captions when needed.The paper reports object inclusion and modification ratios in Table 5, but the supplied passages do not provide their values.Source: §A.5 · pp. 14–15, Table 5 · p. 15

Multiple-choice question evaluation

• Past tasks receive video history from the beginning through the query time, while present tasks use a 60-second sliding window ending at that time.Present-task windows are clamped to start at zero.Source: B. STREAMGAZE Evaluation Details · p. 15
• Past and present tasks are four-way multiple-choice questions, scored by accuracy after deterministic parsing of the model’s answer.Parsing supports concise letters and formats such as “C. pot.”Source: B. STREAMGAZE Evaluation Details · p. 15

Proactive evaluation

• Proactive tasks use cumulative video prefixes at checkpoints around a target’s first fixation, scoring binary decisions on whether it lies within the fixation region.Checkpoints can be 20 or 10 seconds before fixation, at fixation, or 10 or 20 seconds after; the protocol evaluates continuous tracking and alert timing.Source: B. STREAMGAZE Evaluation Details · p. 15
• Proactive tasks model interventions as decisions about when to act on evolving attention or environmental signals, rather than predictions of future events alone.GTA and OAA require monitoring gaze or scene dynamics and generating alerts without an explicit query.Source: B. STREAMGAZE Evaluation Details · p. 15

C.1. Fine-tuning Results

• Fine-tuning data combines external gaze-QA datasets with pipeline-generated examples, including synthesized STREAMGAZE-style data from EgoExoLearn and Ego4D-Gaze.Proactive-task training uses negative examples before first fixation and a positive alert example at the first fixation.Source: §C.1 · p. 16, Figure 8 · p. 16
• Combining all training data sources yields ViSpeak’s best overall accuracy of 0.521, while external gaze-QA datasets alone lower accuracy and performance does not improve uniformly across tasks.EgoExoLearn is particularly helpful for egocentric manipulation scenes; the authors attribute the nonuniform gains to an imbalanced task distribution.Source: §C.1 · p. 16, Table 6 · p. 16

C.2. Details of Gaze Input Prompting

• Text prompting inserts fixation-center coordinates, while salience-map prompting compresses the gaze trajectory into a heatmap prepended to video frames.The figure presents gaze-, text-, and visual-based prompting strategies; the text describes the first two approaches.Source: §C.2 · p. 16, Figure 9 · p. 17, Table 7 · p. 17

C.3. Details of Gaze-based Reasoning

• Qualitative examples show gaze-based reasoning identifying an attended object and visual-based reasoning locating a background item outside the field of view, where text-only reasoning fails.The examples contrast spatially grounded cues with reliance on language priors.Source: §C.3 · p. 17

C.4. Effect of Fixation Filtering

• With rthresh = 0.3 visual degrees, fixation filtering drops 82.3% of Smooth Pursuit and 50.2% of Vestibulo-Ocular Reflex signals.The conservative fixation definition is intended to reduce temporal ambiguity, stabilize intervention timing, and support reproducible comparison; the gaze-only pipeline is more scalable where motion signals are unavailable.Source: §C.4 · p. 17, Figure 10 · p. 18

C.5. Effect of Gaze Signals

• Adding gaze improves performance over no-gaze across tasks, including OI by 10.6%, FAP by 18.2%, and overall by 12.3%; random gaze performs below the no-gaze baseline.The comparison tests models with true gaze, without gaze, and with random gaze.Source: §C.5 · p. 18, Table 8 · p. 18

D.1. Data Statistics

• Videos span cooking, laboratory work, and assembly, with cooking most common; most clips last 5–15 minutes, and extracted FOV objects include action-related tools and manipulable items.Clip lengths range from a few minutes to more than half an hour.Source: §D.1 · p. 18, Table 9 · p. 18
• OAR questions focus on color (65.61%), material (19.3%), state (5.5%), texture (5.3%), and shape (4.1%) attributes of gazed objects.The authors characterize color as the dominant attribute type encountered in egocentric scenes.Source: §D.1 · p. 18

D.2. QA Examples

• Task-specific QA examples are provided in figures, where an orange question mark marks the query time.Source: §D.2 · p. 18

E. Limitations and Future Work

• The benchmark is grounded in manipulation-centric egocentric environments; social interaction, navigation, and outdoor activities remain largely unexplored.Future work could extend coverage to these broader domains.Source: E. Limitations and Future Work · p. 18
• STREAMGAZE’s ten heterogeneous tasks have inherently imbalanced sample sizes, which future work could address through curriculum or task-balanced training.Source: E. Limitations and Future Work · p. 18