PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

Xianyun Sun, Chaoyou Fu, Zhengye Zhang, Feiyang Duan, Qingyuan Cao, Yonghui Niu, Sihang Yuan, Ge Zhang, Caifeng Shan

arXiv:2608.21360v1cs.CV

TL;DR

Interactive video assistants are difficult to evaluate because their responses change subsequent user actions, creating paths that static datasets cannot represent. OmniAssistBench fixes routes with source-video priors and reverse-engineers Internet videos into multi-turn, goal-oriented evaluations. Results expose substantial weaknesses in gesture following, long-term memory, delayed response, and cross-turn context.

  • Problem

    Static offline datasets cannot evaluate interactive assistants whose responses dynamically influence users’ subsequent actions and create diverging paths.

  • Method

    OmniAssistBench derives explicit interaction-path priors from source videos, reverse-engineers Internet footage, and builds Basic, Advanced, and real-world multi-turn tasks.

  • Results

    Gemini-3-Pro reaches 66.4 out of 100, while Qwen3-Omni-Instruct achieves 51.2, with evaluations revealing weaknesses in gesture following, memory, delayed response, and cross-turn context.

  • Takeaways & Limitations

    Current Omni-LLMs generally understand user inputs but remain unreliable on visual prompts, historical context, and waiting for target events.

  • Takeaways & Limitations

    Context capacities are insufficient to preserve complete interaction histories in the benchmark’s multi-turn tasks without specialized long-term-memory mechanisms.

Abstract

from arXiv · show

Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.

1. Introduction

OmniAssistBench addresses the difficulty of evaluating interactive video assistants whose responses alter users’ subsequent actions and create diverging interaction paths. It fixes routes with video-derived priors, reverse-engineers Internet videos into multi-turn interactions, and reveals persistent model weaknesses.

  • Interactive assistants must combine visual state, user goals, and prior knowledge to guide users toward objectives.
  • Static video benchmarks cannot accommodate multi-turn interactions because assistant responses influence users’ subsequent actions and create diverging paths.
  • OmniAssistBench fixes interaction routes by deriving prior knowledge from source videos and requiring models to follow the same procedural path.
  • The benchmark uses Basic and Advanced tiers, plus 3 real-world cases averaging 15 interaction turns, to assess foundational perception and complex goal-oriented abilities.
  • 1000+ expert person-hours were required because scenario design, reverse-engineering, and extensive video editing are labor-intensive.
  • Gemini-3-Pro scores 66.4 out of 100, while Qwen3-Omni-Instruct scores 51.2 under a protocol penalizing redundant and hallucinated information.
  • Evaluations identify weaknesses in gesture following, long-term memory, delayed response, and retaining cross-turn context.

2. Data Construction

OmniAssistBench is constructed through expert-designed scenarios, open-ended multi-turn questions, video editing, embedded prompts, and quality refinement. Video-derived goals and priors preserve intended interaction routes while human annotation supports subtle and goal-dependent judgments.

  • Human annotation remains necessary because subtle video details and user goals are not always recoverable from automated captions.
  • Open-ended multi-turn questions reduce multiple-choice information leakage and better represent interactions without answer options.
  • The four construction phases are scenario design and video collecting, QA design, video editing with prompt embedding, and quality refinement.
  • Experts source footage for detailed plots and collaborate with LLMs to brainstorm scenarios targeting specific model capabilities.
  • Advanced questions use user goals and video-derived priors to enforce a fixed route, such as preparing espresso before milk.
  • Each sample includes a ground-truth sentence and 1–3 key points, with scoring based on semantic similarity and key-point coverage.
  • User prompts are embedded temporally as speech, picture-in-picture visual clips, or subtitles, while videos may be trimmed to fit model context windows.
  • MLLM-based validation adds alternative references when logically correct answers differ substantially in phrasing from the ground truth.

3. Task Design

OmniAssistBench organizes assistant evaluation into foundational perception tasks, advanced goal-oriented capabilities, and three complex real-world cases. Its task design covers visual prompting, context, timing, tracking, and multi-party interaction.

  • Benchmark organization: The benchmark defines 7 major tasks with 16 sub-tasks and supplements them with three real-world cases.The cases address complex scenarios where large-scale data construction is impractical.
  • Typical benchmark tasks: Figure 5 illustrates gesture-based and OCR-based prompt following, context-aware response, multitask tracking, and multi-event-triggered response.These examples represent typical assistant interaction tasks in the benchmark.
  • Basic Interactive Understanding: Basic Interactive Understanding targets foundational perception abilities that are critical in real-world interactions but under-represented in existing video benchmarks.It includes social, temporal, referential, and non-audio prompt-following capabilities.
  • Advanced Interactive Understanding: Advanced Interactive Understanding evaluates actionable, context-grounded advice, delayed responses, and continuous assistance during long-horizon multi-step tasks.Its tasks cover context-aware response, proactive response, and process tracking.
  • Real World Cases: Three real-world cases combine multiple abilities within carefully designed stories: meeting simulation, blind assistance, and handicraft process tracking.The cases are based on one-shot videos and are intended to reflect scenarios more complicated than isolated tasks.
  • Evaluation reporting: The benchmark reports Basic and Advanced task results separately, with overall averages normalized from original 0–5 scores to a 0–100 scale.The same reporting convention is used for the Advanced Interactive Understanding tasks and Real World Cases.

4. Experiments

OmniAssistBench evaluates Omni-LLMs with concurrent video-audio inputs, unified assistant prompting, and rubric-based scoring. Results show that models understand verbal prompts better than they handle visual cues, long-term context, task tracking, and response timing.

  • Evaluation setup: The evaluation suite tests models capable of processing concurrent video and audio streams.
  • Evaluation setup: Scores use an LLM-based structured 5-point rubric normalized to a 100-point scale.The rubric penalizes redundant and hallucinated information.
  • Evaluation setup: 60 is the minimum threshold for a valid response, while scores of 40 and 20 or lower indicate factual failure and fundamental instruction failure, respectively.
  • Overall performance: 66.4 is Gemini-3-Pro’s score, compared with 51.2 for Qwen3-Omni-Instruct, across the reported Basic and Advanced tasks.
  • Basic-tier challenges: Models struggle with non-salient objects, visually similar distractors, and scenes containing more than 10 target objects.
  • Advanced and real-world challenges: Both open-source and proprietary models struggle with gesture instructions, while long context, delayed responses, and linking descriptions to predefined tasks remain additional bottlenecks.Reported context capacities include approximately 380 seconds for MiniCPM-o-2.6 and around 80 seconds for the Qwen family at 1 FPS.
  • Real-world cases: 51.2 is the average closed-source score across three Real World Cases, versus 34.8 for open-source models.The cases include failures in tracking small objects, associating maps with environments, and identifying target people.
  • Real-world cases: Current open-source models remain far from satisfactory real-world assistants, and even the best closed-source models leave substantial room for improvement.

4.3. Ablation Studies

The ablations examine modality dependence, judge consistency, teacher forcing, and video-resolution trade-offs. Results show task-specific modality effects, high judge agreement, limited benefit from corrected histories, and no clear benchmark-wide gain from lower resolution.

  • Input Modality Ablation: Removing either audio or visual input generally reduces performance on most tasks, especially with audio-only inputs.Some visual-only decreases are modest when models already perform poorly on combined audio-video inputs.
  • Input Modality Ablation: All models perform better on Proactive Response with visual-only inputs because background speech can be mistaken for new user instructions.Without distracting audio, models remain focused on the initial prompt.
  • Input Modality Ablation: Gemini-3-Pro performs better with audio-only input on Context-aware Response, partly because its visual-input timestamp–cue format is penalized by the scoring rubric.Audio-only outputs tend to use more colloquial language aligned with spoken responses.
  • Judge Consistency: Pearson correlations between three judge LLMs are close to or above 0.8, indicating strong agreement when scoring candidate models.The judges used identical scoring criteria to evaluate four candidates.
  • Teacher Forcing: Teacher forcing can improve scores by replacing prior model responses with ground-truth answers, but the improvement is not always obvious.Models often adopt the briefer ground-truth format without effectively linking it to the input videos.
  • Input Video Resolution: Reducing Qwen3-Omni-Instruct input resolution provides about 50 seconds of additional temporal context but produces no obvious benchmark-wide score change.The extra context slightly increases scores on most multi-turn questions, yet remains insufficient for videos longer than 10 minutes and may not address every bottleneck.

5. Related Work

Prior video benchmarks increasingly cover streaming, audio-visual, and egocentric settings, but usually evaluate abilities independently through static or single-turn formats. Interactive understanding remains comparatively under-explored, with no comprehensive evaluation pipeline established.

  • Background on Omni-LLMs: Omni-LLMs process video, audio, and text streams concurrently to support real-time human-computer interaction.The field has moved from static and discrete processing toward dynamic multimodal assistants.
  • Video Understanding Benchmarks: Traditional video benchmarks primarily test visual perception with single-turn, multiple-choice questions.Their focus is typically limited to visual content rather than interactive assistance.
  • Video Understanding Benchmarks: Recent benchmarks add streaming, omni-modal, and egocentric videos, but generally evaluate abilities independently using video-question-answer formats.These designs may not directly reflect everyday video-chat performance.
  • Interactive Understanding Evaluation: Interactive understanding benchmarks cover areas such as proactive abilities, streaming perception, proactive reasoning, and associations between visual inputs and user intentions.Existing efforts differ in scope, and LifeEval mainly uses short videos under one minute.
  • Interactive Understanding Evaluation: A comprehensive evaluation pipeline for interactive understanding has yet to be established.The area remains relatively under-explored compared with video understanding.

6. Conclusion

OmniAssistBench evaluates Omni-LLMs as interactive assistants by fixing interaction paths with video-derived priors and simulating interactions through reverse-engineered, edited videos. Experiments identify limitations in gesture following, long-context memory, delayed responses, and cross-turn information maintenance.

  • Conclusion: OmniAssistBench addresses divergent interaction paths by summarizing source-video interaction paths as prior knowledge.Fixing the path enables consistent evaluation of assistant behavior.
  • Conclusion: The benchmark reverse-engineers source videos to design test questions and simulate interactive data through video editing.This annotation pipeline supports offline evaluation of online-style interactions.
  • Conclusion: Experiments highlight limitations in gesture instruction following, long-context memory, delayed response, and maintaining information across turns.These are identified as key limitations of current Omni-LLMs in interactive scenarios.

A. Scoring Rubric

OmniAssistBench scores responses using three dimensions that jointly assess correctness, unwanted content, and compliance with silence requirements.

  • Scoring Dimensions: The scoring rubric evaluates Accuracy and Completeness, Redundancy, and Adherence to Silence Constraints.The rules are provided to the judgment LLM during evaluation.

B. Detailed Task Design

OmniAssistBench organizes its tasks into Advanced Understanding and Basic Understanding sub-tasks. The task descriptions explain the properties used to filter videos during data collection.

  • The task properties described in the tables are used to filter videos during data collection.
  • Table 5 enumerates the Advanced Understanding sub-tasks and provides an italic explanation for each task.
  • Table 6 enumerates the Basic Understanding sub-tasks and provides an italic explanation for each task.

C. Consistence of Different Judge Models

The appendix reports detailed scores from three judge LLMs for four candidate models across evaluated tasks. Results are separated into Basic Interactive Understanding, Advanced Interactive Understanding, and Real World Cases, with scores normalized to 0–100.

  • Three judge LLMs—GPT-5, GLM-5, and DeepSeek-v3.2—scored four candidate models across all evaluated tasks.
  • Basic Interactive Understanding: Table 7 reports candidate-model scores for Basic Interactive Understanding tasks, with Overall Avg. denoting the average across all evaluated tasks.
  • The original 0–5 scores were normalized to a 0–100 scale.
  • Advanced Interactive Understanding and Real World Cases: Table 8 reports candidate-model scores for Advanced Interactive Understanding tasks and Real World Cases, using the same scoring criteria.
Loading 2608.21360v1…