Source-linked AI summary
Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams
Yinsong Yan, Ling Wang, Yuxuan Wang, Yifan Yang, Ruiyang Xu, Junming Lin, Qi Chen, Meng Gao, Minghao Han, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Jibin Wu
TL;DR
Existing evaluations provide limited insight into whether assistants connect stream evidence and user needs to timely, sustained interventions. Omni-InteractScope benchmarks those requirements across 1,479 reviewed interactions, and evaluations reveal limitations in timely tool use, query-free reminders, and complete guidance. The findings support assessing response requirements across an interaction, including responses needed after an initial answer.
Problem
Existing evaluations provide limited insight into how reliably models connect stream evidence and user needs to timely, sustained interventions.
Source: §1 · p. 2Method
Omni-InteractScope organizes 1,479 human-annotated, cross-reviewed interactions with 1,790 temporal targets across eight task groups around modalities, event triggers, intervention types, and user needs.
Source: §1 · pp. 2–3Results
Across 12 models, timely tool invocation, query-free reminders, and complete guidance coverage remain limited; reminder recall never exceeds 23.0%.
Source: §1 · p. 3, §4.2 · p. 7Takeaways & Limitations
Evaluation should assess whether assistants fulfill an interaction’s response requirements, including interventions that remain necessary after an initial answer.
Source: §7 · p. 9Tool-invocation scores alone do not establish event grounding, and reminder coverage depends on response history and frequency.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect stream evidence and user needs to timely, sustained interventions. We introduce \textbf{Omni-InteractScope}, a benchmark of 1,479 human-annotated, cross-reviewed interactions with 1,790 temporal targets across eight task groups. It links input modalities, event triggers, intervention types, and user needs, covering event-timed tool invocation and query-free assistance alongside notifications, warnings, state tracking, and continuous responses. Our experimental results on the proposed benchmark reveal three critical limitations of current Omni-LLMs: (1) \textbf{Timely tool invocation remains unreliable}: all 12 evaluated models produce an on-time first response for at most 40\% of tool-invocation targets, and joint timing-and-tool accuracy reaches at most 32.0\%. (2) \textbf{Query-free reminders remain poorly covered}: all evaluated models achieve lower timely reference-target recall on query-free reminders than on explicit questions, measured on their respective target sets. Reminder recall never exceeds 23.0\%. (3) \textbf{End-to-end guidance coverage remains limited}: nine models cover none of the 26 guidance sequences in full, and the remaining three cover only one to three sequences each. Targeted controls further show that tool-invocation scores alone do not establish event grounding and that reminder coverage depends on response history and frequency. These findings motivate developing and evaluating streaming assistants for timely action, initiative under general user needs, and coverage of entire response sequences.
1 INTRODUCTION
Omni-InteractScope evaluates how assistants connect stream evidence and user needs to interventions that are timely and sustained. Across 12 models, results show limitations in timely tool use, query-free reminders, and complete guidance coverage.
1 INTRODUCTION
- Useful assistance requires connecting perceptual evidence to the needed intervention, its useful timing, and whether further responses remain necessary.The evaluation also distinguishes scene evidence from request priors and target coverage from the opportunities allowed to respond.Source: §1 · pp. 1–2
- Omni-InteractScope contains 1,479 human-annotated, cross-reviewed interactions with 1,790 temporal targets across eight task groups.Targets specify response content and a valid time or interval, including successive responses under continuing user needs.Source: §1 · pp. 2–3
- The benchmark links input modalities, event triggers, intervention types, and user needs to evaluate both individual interventions and continuing obligations.Its scope includes event-timed tool invocation and query-free assistance, alongside notifications, warnings, state tracking, temporal conditions, nonverbal cues, and continuous responses.Source: §1 · pp. 2–3
- Tool invocation tests selecting the appropriate tool and acting when a user-specified event occurs, while query-free assistance tests when a general need warrants intervention.Paired controls examine scene evidence and response policy, and successive targets test whether further responses are delivered as an interaction unfolds.Source: §1 · p. 3
- Across 12 models, on-time first responses occur for at most 40% of tool-invocation targets, while joint accuracy reaches at most 32.0%.Timely reference-target recall on query-free reminders never exceeds 23.0%; nine models cover none of the 26 guidance sequences in full.Source: §1 · p. 3
2 RELATED WORK
Prior benchmarks progress from video comprehension toward streaming response, multimodal interaction, and assistance under general needs. Omni-InteractScope emphasizes user-need-linked interventions, including event-timed action and successive targets.
2 RELATED WORK
- Video-MME and LongVideoBench assess supplied-video comprehension, while StreamingBench and OVO-Bench test responses to unfolding events from available stream prefixes.OmniMMI and OmniPro extend evaluation to multimodal interaction and audio-dependent proactivity, while IPIBench and OmniInteract address evolving requests and turn-taking [Wang et al., 2025b; Zhao et al., 2026; Li et al., 2026; Lu et al., 2026b].Source: §2 · p. 3
- ToolBench and API-Bank evaluate tool-use plans and executable calls, while StreamArena brings tool utilization and proactivity into hour-scale causal streams [Zhang et al., 2026].Omni-InteractScope evaluates event-timed tool identity and invocation timing, retains successive targets, and reports individual-target and complete-guidance coverage.Source: §2 · p. 3
- VizWiz, VIABench, Vinci2, and SoundWatch address visual questions, reminders and guided interaction, assistance without explicit prompts, or environmental-sound relay [Liu et al., 2026; Gong et al., 2026].Omni-InteractScope distinguishes query-free reminders from explicit questions and multi-step guidance, with controls for history and generation opportunities.Source: §2 · pp. 3–4
- Table 1 compares benchmark-level scene-input coverage with interaction objectives.Its notes distinguish audiovisual input from audio necessity and define query-free tasks and successive targets.Source: Table 1 · p. 4, §2 · p. 4
3 OMNI-INTERACTSCOPE
Omni-InteractScope treats an interaction as a stream, user context, and one or more required interventions. It organizes eight task groups around response content, timing, and continuing obligations, using human annotations reviewed by separate reviewers.
3 OMNI-INTERACTSCOPE
- Omni-InteractScope comprises 1,479 interactions across eight task groups, organized around the interventions an assistant should deliver as a stream unfolds.Its benchmark unit retains one or more reference interventions within a video and audio stream and user context.Source: §3 · p. 4, §3.1 · p. 4
3.1 DESIGNING FOR TIMELY AND SUSTAINED INTERVENTION
- The design links available modalities, event triggers, intervention types, and user needs to evaluate response content, timing, and coverage.Each target specifies the required response and its reference time or valid interval; continuing requests retain successive targets in temporal order.Source: §3.1 · p. 4
3.2 TASK COVERAGE AND FOCAL SETTINGS
- The task groups cover event notifications, risk warnings, workflow cues, cumulative-state tracking, temporal conditions, continuous responses, tool invocation, and impairment assistance.T1 tests tool identity and invocation time but excludes tool arguments and downstream execution; T8 distinguishes query-free reminders, explicit questions, and multi-step guidance.Source: §3.2 · pp. 4–6
3.3 DATA CONSTRUCTION AND ANNOTATION
- The benchmark combines 905 online video records for explicit-request tasks with 207 self-recorded first-person records for impairment assistance, and includes English and Chinese QA text.Human annotators specified context, reference responses, triggers, and response times or valid intervals; a separate reviewer group cross-checked the annotations.Source: §3.3 · p. 6
4 EXPERIMENTS
The experiments evaluate 12 multimodal model configurations on explicit-request and assistance tasks, scoring whether responses match required content and timing. Results show uneven task performance, lower coverage for query-free reminders than explicit questions, and a gap between guidance-step and full-sequence coverage.
4.1 EXPERIMENTAL SETUP
- For response j, joint eligibility requires its timestamp to fall within the target window and its content to be correct for the user context.T1 requires an exact tool-name-set match; T2–T8 use a binary GPT-4o judge for content, with timing checked separately.Source: §4.1 · p. 6
- T1–T7 score the first response assigned to each target, while T8 matches responses to targets in order and uses each response at most once.A premature or incorrect T1–T7 response cannot be rescued by a later correction.Source: §4.1 · p. 6
4.2 MAIN RESULTS
- After media-validity exclusions, the evaluation covers 1,477 interactions and 1,788 targets; Table 3 pools assistance hits across T8-A, T8-B, and T8-C.Table 3 includes 122 reminder, 151 audiovisual-QA, and 87 guidance-step targets; complete T8-C counts full sequences out of 26.Source: §4.2 · p. 7, Table 3 · p. 8
- The best scores for cumulative-state tracking (T4) and continuous responses (T7) are only 11.2% and 11.4%, while Gemini-3.5-Flash leads the six-task average and Qwen3.5-Omni-Plus leads tool invocation.Gemini-3.5-Flash responds prematurely on 172/206 T4 targets; its T7 failures include 233/457 premature responses and 152/457 timely content failures.Source: §4.2 · p. 7
- Every configuration covers a smaller fraction of query-free reminders than explicit questions; Gemini-3.1-Pro reaches 23.0% on reminders versus 62.3% on questions.The reminder and question results use different event sets.Source: §4.2 · p. 7
- Gemini-3.1-Pro covers 51.7% of T8-C guidance steps but completes only 3 of 26 sequences; no open-source prompted or streaming configuration completes one.Complete T8-C measures annotated-trajectory coverage, not closed-loop user-task success.Source: §4.2 · p. 7
5 DIAGNOSTIC ANALYSIS
The diagnostic analyses test whether tool scores reflect scene evidence, how response policies affect reminder coverage, and how step coverage relates to full guidance completion. They find that request priors and response opportunities matter, while step success does not guarantee complete sequences.
5.1 SCENE EVIDENCE AND TOOL-INVOCATION SCORES
- Removing scene input does not lower Gemini-3.5-Flash’s T1 accuracy: joint accuracy is 29.50% with scene input and 33.00% without it, a paired difference of −3.50 percentage points.Requests and evaluation protocol are held fixed, so joint accuracy alone does not establish event-grounded invocation.Source: §5.1 · p. 8
- Gemini-3.5-Flash selects 96.5% of tools from request text and public definitions alone; a fixed three-second delay yields 68.0% joint success in a separate control.The control uses a closed-label interface and different response opportunities, diagnosing request and timing priors rather than a matched model ranking.Source: §5.1 · p. 8
5.2 REMINDER COVERAGE UNDER DIFFERENT RESPONSE POLICIES
- Adding response history cuts output volume by 58.1% while both conditions cover 22 of 122 T8-A targets; adding a cooldown costs 9.84 percentage points of recall versus history alone.History changes which targets are covered, with 12 gained and 12 lost; cooldown skips generation calls and restricts coverage opportunities.Source: §5.2 · p. 8
- Event-specific instructions benefit Gemini-3.5-Flash and Qwen2.5-Omni when responses are scheduled at valid times, but effects under autonomous responding are small and uncertain.Scheduling changes temporal information and response opportunities together, so the comparison does not isolate a planning mechanism.Source: §5.2 · p. 8
5.3 COMPLETE GUIDANCE RELATIVE TO STEP COVERAGE
- Gemini-3.1-Pro covers 30 of 60 steps but completes only three of 20 three-step T8-C sequences; an independence-based reference expects 2.4 completions, consistent with the observed count.The comparison does not establish a separate memory or planning failure.Source: §5.3 · pp. 8–9
6 DISCUSSION
The discussion argues that tool scores, reminder recall, and sequence completion depend on controls and response policies, so evaluation should distinguish grounding, coverage, and policy effects.
6 DISCUSSION
- T1 controls show request information and response timing can sustain joint tool-and-time success without scene inputs, so tool scores need evidence controls to assess event grounding.Varying event occurrence and timing while holding requests fixed is proposed as a stronger test for streaming tool use.Source: §6 · p. 9
- Response history reduces Gemini-3.5-Flash’s output volume with the same aggregate recall point estimate, while a cooldown reduces both outputs and coverage.The discussion recommends reporting history access and output rate alongside recall; selectivity also requires evaluating intervention value and cost beyond annotated targets.Source: §6 · p. 9
- Complete T8-C measures whether users receive every guidance step, while the sequence reference limits attributing low completion to a separate memory or planning failure.Records of delivered guidance and pending obligations are presented as a design hypothesis to test through interventions on history and state.Source: §6 · p. 9
7 CONCLUSION
The conclusion presents Omni-InteractScope as a shared benchmark for timely and sustained intervention, and the appendix guide directs readers to details on data, scoring, model settings, controls, and examples.
7 CONCLUSION
- Omni-InteractScope links user needs and stream evidence to 1,790 timed response targets across eight task groups, and evaluation across 12 models finds limitations in timely invocation, reminders, and complete guidance coverage.The paper argues assistants should be assessed on response requirements that may remain after an initial answer.Source: §7 · p. 9
- Generative AI supported taxonomy reorganization, translation, manuscript editing, preliminary validation, and qualitative-case presentation, while benchmark media and annotations underwent human review.The authors state AI was not used to generate synthetic data, design experiments, implement evaluated methods, or clean the dataset.Source: AI USE STATEMENT · p. 10
APPENDIX GUIDE
- Appendix A covers data and annotation, B observation and scoring, C model settings and prompts, D controls and sensitivity analyses, and E illustrative model responses.Sections 3 and 4.1 describe benchmark design, construction, experimental setup, and primary metrics.Source: REPRODUCIBILITY STATEMENT · p. 10, APPENDIX GUIDE · p. 13
A.1 INTERACTION RECORDS AND TASK COVERAGE
The appendix specifies how interaction records link user needs and stream evidence to ordered, timed response targets. It defines task boundaries and the evaluation procedures used to assess individual interventions and continuing assistance.
A.1 INTERACTION RECORDS AND TASK COVERAGE
- 1,479 interactions contain 1,790 targets; multiple targets occur in 163 interactions, and Table 4 partitions targets by task and annotated trigger modality.Evaluation uses 1,477 interactions and 1,788 targets, including a separate 115-interaction T5 diagnostic.Source: §A.1 · p. 13, Table 4 · p. 13
A.2 TAXONOMY DEFINITIONS AND BOUNDARIES
- Tasks distinguish event-triggered tools, notifications and warnings, accumulated-state or gesture cues, query-free assistance, explicit questions, and continuing response obligations.T1 scores tool identity and timing, not arguments or execution; T8-A requires reminders without event-specific questions, while T8-C covers successive guidance steps.Source: §A.2 · pp. 13–14
A.3 SOURCES AND ANNOTATION
- Each interaction records the request or general need, triggering evidence, target content, and a target time or valid interval; related targets remain ordered.A separate reviewer group cross-checks annotations, and the 1,112 video records are storage units rather than necessarily distinct original sources.Source: §A.3 · p. 14
A.4 ANNOTATION CONSISTENCY CHECKS
- A pre-release audit found evidence support in 62/64 cases, timing in 63/64, modality in 62/64, and tool or assistance need in 63/64.The audit included 64 primary and 16 secondary reviews; five cases had Gemini-3.5-Flash-assisted review with human confirmation.Source: §A.4 · p. 14
A.5 FROM SCENE EVIDENCE TO AN ANNOTATION RECORD
- Figure 4 represents a cumulative-state target using selected frames, an annotated request and response, and a 48-second point target.The evaluator applies a two-second tolerance around the point; this is not an additional human-annotated interval.Source: Figure 4 · p. 15, §A.5 · p. 15
B.1 OBSERVATION SCHEDULES AND RESPONSE HISTORY
- Main-task decisions use query time, available integer ticks, and clip duration; models receive neither reference answers nor target times.Prompted models make independent prefix calls, whereas stateful adapters retain a session initialized by replaying prior observations; initialization outputs are excluded.Source: §B.1 · p. 15
B.2 NOTATION AND TEMPORAL ELIGIBILITY
- Targets specify required content and valid time windows; content and timing are checked separately, with content judged against the textual reference.T1–T7 use a ±2-second window around point targets, while T8-A uses its annotated interval without extension; T8-B and T8-C apply a query-time lower bound.Source: §B.2 · p. 16, §B.3 · p. 16
B.4 FIRST-RESPONSE ASSIGNMENT FOR T1–T7; B.5 ORDERED MATCHING FOR ASSISTANCE
- T1–T7 assign each target its earliest eligible response after a midpoint lower bound, consuming it even if premature or incorrect; assistance matching instead maximizes ordered eligible pairs.Assistance matching permits a later eligible response after an earlier failed attempt, and unmatched outputs incur no penalty, so target recall measures coverage rather than false-alarm rate or intervention cost.Source: §B.4 · p. 17, §B.5 · p. 17
B.6 TASK AGGREGATES AND COMPLETE-SEQUENCE COVERAGE
The section defines task-level aggregates and complete-sequence coverage, then specifies paired uncertainty estimates and response-assignment rules. Together, these rules distinguish target success from timing, assignment, and full-sequence coverage.
B.6 TASK AGGREGATES AND COMPLETE-SEQUENCE COVERAGE
- Task accuracy for T1–T7 and target recall for T8 use pooled successes over targets; Table 2’s Avg. equally weights six task accuracies.Table 3’s Overall pools 361 T8 targets, while T5 is a separate rolling diagnostic marked N/A for prompted inputs.Source: §B.6 · pp. 17–18
- Complete T8-C counts interactions with every guidance step covered out of 26, separating full annotated-trajectory coverage from isolated correct steps.It does not measure closed-loop task completion.Source: §B.6 · p. 18
B.7 PAIRED UNCERTAINTY ESTIMATES
- Paired differences use 95% percentile intervals from 10,000 video-record-cluster bootstrap resamples, excluding judge error and API run-to-run variation.Zero-width intervals indicate constant observed paired differences, not population equivalence; intervals have no multiplicity correction.Source: §B.7 · p. 18
B.8 WORKED EXAMPLES OF RESPONSE ASSIGNMENT
- T1–T7 score the first response, so an early response cannot be replaced by a later eligible one; T8 instead matches eligible responses to ordered targets.A response can match at most one target; unmatched outputs do not reduce recall and are reported alongside output volume.Source: §B.8 · p. 19
C EXPERIMENTAL DETAILS
The experimental details specify causal audiovisual inputs, model configurations, prediction and scoring prompts, and diagnostic controls. The tool-invocation results show that timely action and correct tool selection remain distinct challenges.
C.1 INPUT MODALITIES AND CONTEXT
- At each decision time, prompted models receive video frames only from the available prefix, while audio and transcript inputs likewise follow causal-prefix or completed-chunk policies.Video is sampled at 1 fps, capped at 32 frames; V+ASR does not necessarily transcribe the whole prefix or non-speech sounds.Source: §C.1 · pp. 19–20, Table 6 · p. 20
C.2 MODEL SIZES AND GENERATION SETTINGS
- Model sizes and generation settings vary across configurations, and context retention and computation are not equalized across interfaces.Proprietary parameter counts are undisclosed.Source: §C.2 · p. 20
C.3 CORE PREDICTION AND SCORING PROMPTS
- Prompts distinguish explicit-request tool decisions from query-free monitoring, answering, and stepwise guidance; the scheduled control supplies timing but not an event or answer.T8 assistance is constrained to current causal scene evidence and the user profile or request.Source: §C.3 · p. 21
- The content judge checks semantic correspondence to a target and essential action, while timing and ordered assignment are handled separately by the scorer.A generic warning does not cover a specific target; language compliance is not added as a reported metric.Source: §C.3 · p. 22
D.1 TIMELY ACTION: TOOL CHOICE AND EVENT TIMING
- MOSS-VL selected the correct tool set in 3 of 35 timely responses, while LiveCC had 80 timely responses and no joint successes.The results apply to the evaluated model–adapter configurations.Source: §D.1 · p. 22, Table 7 · p. 23
D.1.1 REQUEST AND TIMING PRIORS
- Gemini-3.5-Flash recovered 193/200 tool labels from request text alone, but this did not identify the search referent in the stream.Its descriptive maximum among tested fixed delays was 68.00% at three seconds, without independent development-set selection.Source: §D.1.1 · pp. 22–23, Table 8 · p. 23
D.1.2 PAIRED SCENE ABLATION
The paired scene ablation compares Gemini-3.5-Flash with and without scene input, while a separate control contrasts event-specific versus general guidance under scheduled and autonomous responding.
D.1.2 PAIRED SCENE ABLATION
- Scene-present joint scoring was 3.50 percentage points lower than scene-removed scoring (95% CI: [−7.00, −0.50], 200 clusters).The separate runs had 59 versus 66 joint hits; the main Gemini-3.5-Flash row had 59 hits and 75 on-time responses.Source: §D.1.2 · pp. 23–24
- The scene ablation measures input-removal effects but does not establish counterfactual tracking of event occurrence or timing.Scene-present and scene-removed runs had 123 premature responses each.Source: §D.1.2 · pp. 23–24
D.2.1 EVENT GUIDANCE AND SCHEDULED RESPONSES
- Only Qwen2.5-Omni showed a statistically supported guidance-by-scheduling interaction; scheduling changes temporal information and response opportunities together.On the reachable subset, its interaction was +6.72 points (95% CI: [+1.71, +12.00]); this does not isolate a planning mechanism.Source: §D.2.1 · p. 24, Table 9 · p. 24
D.2.2 RESPONSE HISTORY AND RESPONSE COOLDOWN
The diagnostics examine how response history, output filtering, decision schedules, scoring tolerance, task type, and audio inputs affect measured coverage. They also distinguish step coverage from complete guidance and identify limits on interpreting these comparisons.
D.2.2 RESPONSE HISTORY AND RESPONSE COOLDOWN
- Response history reduced Gemini-3.5-Flash outputs by 58.09% while aggregate coverage remained 22/122; adding a ten-second cooldown reduced coverage to 10/122.History and cooldown conditions changed output opportunities, and the history comparison does not establish equivalence.Source: §D.2.2 · pp. 24–25, Table 10 · p. 25
D.2.3 FILTERING RECORDED OUTPUTS
- Five-second spacing left Gemini-3.5-Flash with 11 of 27 target matches and 1,642 of 7,813 outputs under general instructions.For Qwen3-Omni, the same spacing left two of three matches and 1,666 of 8,213 outputs; filtering does not change subsequent model behavior.Source: §D.2.3 · p. 25
D.3 SUSTAINED COVERAGE: STEPS AND COMPLETE GUIDANCE
- Gemini-3.1-Pro covered 51.7% of T8-C steps but completed only 3 of 26 guidance sequences; complete coverage measures the annotated trajectory, not closed-loop task success.Its three-step episodes had 8/20, 10/20, and 12/20 covered steps, with three observed completions within the permutation reference interval.Source: §D.3 · pp. 25–26, Figure 6 · p. 26
D.4.1 DECISION SCHEDULES
- Gemini-3.5-Flash accuracy fell by 7.63 points under continuous rather than target-local decisions as premature responses rose from 58.45% to 66.90% of targets.First-response scoring exposes early actions that target-local probing cannot observe.Source: §D.4.1 · pp. 26–27, Table 11 · p. 26
D.4.2 TEMPORAL-TOLERANCE SENSITIVITY
- Across one-, two-, and four-second tolerances, all scene-present minus scene-removed point estimates were negative, but the one-second interval included zero.Score magnitude and statistical support were not invariant; the sensitivity result is specific to the Gemini-3.5-Flash control.Source: §D.4.2 · p. 27, Table 12 · p. 27
D.4.3 TIMING AND CONTENT FAILURES
- Gemini-3.5-Flash’s T4 failures were predominantly premature (172/206 targets), while T7 had premature responses (233/457) and timely content failures (152/457).The breakdown cannot infer the content correctness of premature responses.Source: §D.4.3 · p. 27, Table 13 · p. 27
D.4.4 SCENE AUDIO WITH THE REQUEST HELD FIXED
- Native audio added seven hits on 78 auditory targets for both Gemini-3.5-Flash and Qwen3-Omni, while Gemini-3.5-Flash gained only 0.29 points overall.These runs used a separate schedule and earlier verdict handling; the results do not prove audio is necessary for every target.Source: §D.4.4 · pp. 27–28, Table 14 · p. 28
E.1 TIMELY ACTION AND QUERY-FREE REMINDERS
These selected cases show that assistance can miss the requested moment or lose consistency as the stream changes, and that partial responses may still leave a guidance sequence incomplete. The examples illustrate failure modes, not their prevalence.
E.1 TIMELY ACTION AND QUERY-FREE REMINDERS
- Gemini-3.5-Flash called the tool before the scored window, while all five in-window Gemini-3.1-Pro replies recommended exiting instead of waiting.The selected cases illustrate failure modes, not their prevalence, and do not establish an internal cause.Source: Figure 7 · p. 29, §E.1 · p. 29
E.2 CODE DIAGNOSIS UNDER CHANGING SCREEN STATES
- Gemini-3.1-Pro gave locally grounded C# advice, then claimed visible initializer braces had been deleted.The Figure 8 diagnoses illustrate screen agreement but are not additional scored targets; useful earlier advice does not establish later consistency.Source: §E.2 · p. 30, Figure 8 · p. 30
E.3 A TUTORIAL REMINDER AT THE REQUESTED STAGE
- Qwen3-Omni responded at 162.00 s before the requested tutorial stage, while Gemini-3.1-Pro matched at 191.00 s.Figure 9 marks the annotated second-method point at 189.87 s with a ±2-second window, distinguishing a plausible operation description from stage-timed assistance.Source: §E.3 · p. 31, Figure 9 · p. 31
E.4 PARTIAL COVERAGE WITHIN A GUIDANCE SEQUENCE
- 2/3 targets were covered, but none of the guidance sequence was complete despite 67 outputs.All four final-target candidates were incorrect; ordered assignment allows an incorrect 21-second attempt not to prevent a later 22-second match.Source: §E.4 · p. 32, Figure 10 · p. 32
Related papers
This paper cites
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs2023
- VizWiz Grand Challenge: Answering Visual Questions from Blind People2018
- OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding2026
- Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos2026
- Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction2026
- OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs2026