PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

Sunqi Fan, Qingle Liu, Runqi Yin, Meng-Hao Guo, Shuojin Yang

arXiv:2606.29445v1cs.CVcs.AI

TL;DR

Existing VideoQA benchmarks provide limited evidence that MLLMs can learn procedural knowledge from tutorials and transfer it to long-horizon GUI tasks. This paper introduces VG-GUI-Bench and TASKER, a task-driven, scene-aware keyframe search method, and reports consistent gains across VideoQA and video-guided agentic benchmarks, including a 2.0% improvement over the best baseline on EgoSchema fullset.

  • Problem

    Existing VideoQA evaluations focus on shallow perceptual cues, providing limited evidence that MLLMs can learn procedural knowledge and generalize it to long-horizon tasks.

  • Method

    The paper introduces VG-GUI-Bench for tutorial-to-GUI procedural transfer and TASKER, a task-driven, scene-aware graph-search algorithm for extracting informative keyframes.

  • Results

    TASKER consistently improves accuracy and frame efficiency across VideoQA and video-guided agentic benchmarks, surpassing the best baseline by 2.0% on EgoSchema fullset.

  • Takeaways & Limitations

    Generalized keyframe extraction can bridge VideoQA and video-guided agentic tasks while improving accuracy and frame efficiency.

  • Takeaways & Limitations

    TASKER-DFS is prone to local optima without strict termination conditions, so the paper does not introduce that variant.

Abstract

from arXiv · show

Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks. However, existing benchmarks primarily evaluate whether models can perceive shallow visual cues, while rarely examining whether MLLMs can learn deeper knowledge or procedural skills from video tutorials and generalize them to downstream long-horizon agentic tasks. To address this gap, we introduce VG-GUIBench (Video-Guided GUI Benchmark), a new benchmark designed to evaluate whether MLLM-based GUI agents can follow video tutorials to complete corresponding GUI interactive tasks. Furthermore, we observe that the performance of models on both VideoQA and video-guided agentic tasks critically depends on effective keyframe extraction. Based on this observation, we propose TASKER (Task-driven And Scene-aware Keyframe searchER), a keyframe extraction algorithm that jointly considers task relevance and scene dynamics to identify informative frames. Experimental results demonstrate that TASKER achieves significant performance improvements on both VideoQA and video-guided agentic task benchmarks, outperforming the best baseline by 2.0% on the EgoSchema fullset and 1.8% on the NExT-QA dataset, respectively. These results further highlight the potential of generalized keyframe extraction methods for video understanding tasks. Our code and data are available at https://github.com/VG-GUI-TASKER/VG-GUI-TASKER.

1 Introduction

The paper argues that VideoQA benchmarks undermeasure whether MLLMs acquire procedural knowledge and transfer it to long-horizon GUI tasks. It introduces VG-GUI-Bench and TASKER to evaluate and improve this progression from video perception to action.

  • Motivation: Existing VideoQA benchmarks emphasize shallow cues, leaving unclear whether MLLMs can learn deeper knowledge and procedural skills for long-horizon agentic tasks.Tutorial-based tasks require step-by-step procedure understanding, abstraction of key operations, and transfer beyond visual-event recognition.
  • Taxonomy: The paper frames VideoQA and video-guided agentic tasks as progressive levels from perception to action, with the latter requiring procedural learning and long-horizon execution.VideoQA involves temporally relevant evidence and question-conditioned reasoning, whereas agentic tasks transfer demonstrated procedures into GUI actions.
  • Benchmark: VG-GUI-Bench pairs tutorial videos with semantically related GUI tasks to test whether MLLM agents transfer procedural knowledge to new interactive tasks.The benchmark provides a practical testbed for video in-context learning.
  • Method: TASKER combines task-driven relevance estimation with scene-aware temporal dynamics to select compact, informative keyframes while discarding redundant content across both task types.The method addresses a shared bottleneck: redundant long-video segments and briefly appearing procedural evidence make naive sampling miss critical moments or degrade reasoning.
  • Results: 63.1% accuracy on EgoSchema fullset and 77.4% average accuracy on NExT-QA surpass the best baselines by 2.0% and 1.8%, respectively.TASKER also achieves high frame efficiency and consistently outperforms prior temporal-selection and video-agent methods, including VideoTree and VideoAgent.

2 Related Work

Related work spans VideoQA, keyframe extraction, and video-guided tasks. VideoQA has evolved from CNN-based encoders and lightweight fusion toward visual encoders integrated with projection layers and LLMs, while keyframe research explores diverse selection strategies and video-guided benchmarks study learning skills from instructional videos.

  • Video Question Answering: VideoQA evaluates reasoning over temporal visual content in response to language queries, with recent benchmarks emphasizing long-form videos and complex reasoning.
  • Video Question Answering: VideoQA methods progressed from CNN-based visual encoders with lightweight fusion to pretrained visual encoders combined with projection layers and LLMs [27] [45] [54] [71].
  • Keyframe Extraction: Keyframe extraction methods include learned frame selection, attention-based segment extraction, uniform sampling with image-grid modeling, recursive agent-guided selection, and VideoTree’s hierarchical feature-clustering search.
  • Video-Guided Tasks: Video-guided tasks evaluate acquiring tasks or skills from instructional videos [19] [20] [63] [66], including GUI operation knowledge transfer through Mobile-Agent-V [50].
  • Video-Guided Tasks: The field has introduced datasets [32] [33] and benchmarks [7] [8] [30] for evaluating video-guided task learning.

3 Method

Section 3 introduces VG-GUI-Bench, a long-horizon benchmark for evaluating MLLM agents on video-guided GUI tasks, and TASKER, a task-driven keyframe search method that selects sufficient visual information for answering questions or completing agentic tasks.

  • VG-GUI-Bench: VG-GUI-Bench systematically evaluates MLLMs on long-horizon GUI task execution from tutorial videos using task-specific prompts and an average of 10.71 steps per episode.The benchmark builds on MON-DAY and provides tutorial videos, ground-truth action sequences, and keyframe screenshots as references.
  • VG-GUI-Bench: Its standardized action space defines six GUI operations: CLICK, SCROLL, TYPE, PRESS, ZOOM, and FINISH.PRESS supports BACK, HOME, and ENTER, while CLICK and SCROLL use coordinate arguments; ZOOM and FINISH are argument-free.
  • VG-GUI-Bench: The evaluation iteratively selects video keyframes, prompts the MLLM to predict the next action, executes it, and compares the resulting sequence with ground-truth actions.VG-GUI-Bench reports accuracy, completion, efficiency, and Performance Improvement Rate, measuring action correctness, executed-step proportion, frame cost, and the video benefit respectively.
  • TASKER: TASKER searches a tree of video segments for a sufficient keyframe set, using visible frames either directly in the MLLM or through generated captions before predicting an answer or action.The initial node is the entire video, and segment expansion continues toward information needed to answer the question or complete the task.
  • TASKER: TASKER-GBFS uses MLLM-estimated missing visual information as a task-driven cost, whereas TASKER-BFS expands segments breadth-first without MLLM cost evaluation.The method does not introduce a DFS variant because single-segment expansion can fall into local optima without strict termination conditions.

4 Experiments

TASKER outperforms competing keyframe-selection and VideoQA methods while using fewer visible frames, and it achieves strong performance on video-guided GUI tasks. Ablations show that its search, termination, and base-LLM choices materially affect accuracy and efficiency.

  • VideoQA: TASKER with GPT-4 achieves 63.1% accuracy on the EgoSchema fullset, surpassing the best baseline by 2.0% and outperforming all compared VideoQA methods.The comparison uses the TASKER-A* variant in a training-free, zero-shot setting, which also outperforms training-based methods such as LVNet and Vamos.
  • Frame efficiency: At 66% accuracy on an EgoSchema subset, TASKER uses about one-quarter as many frames as VideoTree while accessing only visible frames.Reported performance requires about 15% of total video frames, whereas LangRepo and LifelongMemory process all frames without selection.
  • VG-GUI-Bench: On VG-GUI-Bench, adding 10 uniformly sampled frames improves performance for most models over no video, while Gemini-3.1-Pro achieves the best overall accuracy in both settings.Seed-2.0-Pro shows the largest gain, increasing from 35.93% to 39.78%.
  • VG-GUI-Bench: TASKER-A* achieves the highest Overall Acc. (40.96), PIR (0.618), and CLICK score (53.68) on VG-GUI-Bench, outperforming VideoTree.TASKER also deduplicates redundant frames, achieving higher accuracy with fewer frames per step than other dynamic selection methods.
  • Ablations: TASKER-A* achieves the best variant performance, while GPT-4o is the strongest base LLM and outperforms o3-mini and Deepseek-R1.The search-algorithm ablation evaluates visible-frame efficiency on three-minute EgoSchema videos initialized at fps = 1, or 180 frames.
  • Ablations: Combining self-evaluation, self-reflection, and temporal summarization improves termination confidence and algorithm performance by assessing information sufficiency from complementary perspectives.The termination-condition ablation uses TASKER-A* on the EgoSchema subset.

5 Analysis

The analysis highlights TASKER’s efficiency and interpretability advantages over end-to-end Video-LLMs, while a visual case study shows that it can precisely locate question-relevant video content. In the example, TASKER retrieves the critical 125s–130s segment and successfully answers the question.

  • Efficiency and interpretability: TASKER offers lower training cost, greater inference efficiency, and better interpretability than end-to-end Video-LLMs.A detailed comparison of performance and resource requirements is provided in Appendix C.
  • Case study: In a 3-minute video, TASKER precisely identifies the critical information segment between 126s and 130s.Figure 5 presents this visualized case study.
  • Case study: TASKER searches along the video tree, expands relevant nodes, and retrieves all frames from 125s–130s as keyframes, successfully answering the question.The video tree marks nodes traversed by the key search path in yellow.

6 Conclusion · Appendix

The work connects VideoQA with higher-level video-guided agentic tasks through VG-GUI-Bench and TASKER. VG-GUI-Bench evaluates procedural-knowledge transfer to long-horizon decision making, while TASKER treats keyframe extraction as a generalized graph-search problem.

  • 6 Conclusion: The work explicitly connects low-level VideoQA with higher-level video-guided agentic tasks as a step toward deeper video understanding.
  • 6 Conclusion: A visualization illustrates TASKER’s tree-search and node-expansion process on an EgoSchema VideoQA case [34].
  • 6 Conclusion: VG-GUI-Bench pairs tutorial videos with corresponding GUI interaction episodes to evaluate whether MLLM-based agents transfer procedural knowledge to long-horizon decision making.
  • 6 Conclusion: TASKER is a task-driven and scene-aware keyframe-search algorithm built around the shared bottleneck of temporal content selection across VideoQA and video-guided agentic tasks.
  • 6 Conclusion: TASKER formulates keyframe extraction as a generalized graph-search problem for deeper video understanding.

A Preliminary: Classic Search Algorithm

TASKER is built on a classic search procedure that repeatedly selects nodes by priority, checks for the destination, expands neighbors, and continues until success or exhaustion. DFS, BFS, and GBFS differ in how they prioritize nodes during this process.

  • Classic Search Algorithm: TASKER builds on a classic search algorithm that initializes an open list, repeatedly selects a prioritized node, expands its neighbors, and stops upon reaching the destination or exhausting the list.The procedure returns the destination node when found and otherwise returns None after the search list becomes empty.
  • Classic Search Algorithm: DFS prioritizes greater-depth nodes, whereas BFS explores all neighbors at the current level before advancing to the next level.DFS searches deeply before backtracking, while BFS proceeds level by level.
  • Classic Search Algorithm: GBFS prioritizes nodes using a heuristic h(n) as its cost function, f(n) = h(n), guiding search toward the destination without guaranteeing an optimum.Here, h(n) denotes the cost from the current node to the destination.

B Benchmarks

The benchmarks evaluate long-form video understanding through challenging question-answering datasets, with EgoSchema emphasizing extended videos and NExT-QA covering causal, temporal, and descriptive reasoning.

  • EgoSchema: EgoSchema contains over 5,000 human-curated multiple-choice QA pairs, including a 500-video subset for long-form video question answering.Each video is three minutes long, and humans achieve only 76% accuracy while current Video-LLMs remain below 70%.
  • NExT-QA: NExT-QA comprises 5,440 videos and approximately 52K manually annotated QA pairs for evaluating video understanding.Its questions include causal reasoning about motivations or outcomes, temporal reasoning about action sequence and timing, and descriptive reasoning about scenes, objects, attributes, and events.

C Comparison with Video-LLMs

The section contrasts end-to-end Video-LLMs with TASKER’s training-free, keyframe-based approach, emphasizing a tradeoff between accuracy and computational efficiency. TASKER further reduces inference overhead and improves interpretability through explicit intermediate outputs.

  • State-of-the-art Video-LLMs [4] outperform keyframe-based, training-free methods, making them suitable when high accuracy matters and computational cost is acceptable.
  • TASKER’s training-free design reduces overall cost compared with Video-LLMs requiring substantial training resources for comparable EgoSchema and NExT-QA performance.Table 7 compares the training costs and resource requirements of comparable Video-LLMs, highlighting their complexity and expense.
  • TASKER lowers inference overhead by selecting efficient keyframes instead of processing the entire video, while still relying on large-model inference.
  • TASKER improves interpretability by exposing keyframe selections and textual reasoning as intermediate results, unlike end-to-end Video-LLMs.These intermediate outputs enhance transparency and interpretability.

D Implementation Details · E Detailed Demonstration of VG-GUI-Bench

The implementation uses benchmark-specific captioners and configurable TASKER sampling, while VG-GUI-Bench is demonstrated through a multi-step GUI task evaluated with tutorial keyframes as visual guidance. Together, these details specify both the video-understanding setup and the agent-interaction evaluation pipeline.

  • D Implementation Details: For VideoQA, the system captions visible frames before using LLM reasoning, with CogAgent [18] for NExT-QA and LaViLa [73] for EgoSchema.LaViLa is selected for EgoSchema because of its egocentric video pretraining, following VideoAgent.
  • D Implementation Details: The base LLM versions are gpt-4-1106-preview and gpt-4o-2024-11-20.These are the specific model versions used in the implementation.
  • D Implementation Details: TASKER’s visible-frame count depends on the initial segment count M and maximum search iterations T; the main experiments use M = 10 and T = 6.Changing these parameters affects final accuracy.
  • E Detailed Demonstration of VG-GUI-Bench: VG-GUI-Bench evaluates whether an agent can execute multi-step GUI instructions, illustrated by saving emails as PDF on an iOS device.The case study is presented in Figure 6 as an intuitive demonstration of the benchmark.
  • E Detailed Demonstration of VG-GUI-Bench: At each interaction step, the agent receives the current GUI frame, previous actions, and TASKER-extracted tutorial keyframes as visual guidance.These inputs form the stepwise evaluation pipeline described for the benchmark.
  • E Detailed Demonstration of VG-GUI-Bench: The MLLM predicts an action type such as CLICK or SCROLL and corresponding arguments such as screen coordinates.The supplied passage describes prediction of both the action category and its parameters.

F Prompts

The appendix specifies TASKER’s prompt-driven keyframe search and self-evaluation procedures, then defines VG-GUI-Bench evaluation prompts across no-video, oracle-keyframe, and algorithm-selected-keyframe settings.

  • F.1 TASKER Keyframe Selector Prompts: TASKER prompts MLLMs to assess candidate video gaps and determine whether the current keyframes sufficiently cover the task.The prompt suite includes BFS, GBFS, Dijkstra, A*, and QA & Reflect strategies.
  • F.1 TASKER Keyframe Selector Prompts: BFS explores multiple gaps simultaneously for missing UI actions and returns segment descriptions in a prescribed JSON structure.The strategy encourages selecting several segments when different gaps may contain important missing actions.
  • F.1 TASKER Keyframe Selector Prompts: GBFS selects one segment most likely to contain missing goal-critical actions, focusing on unexplained transitions in the operation flow.Its output identifies a single segment and describes it as containing missing goal-critical actions.
  • F.1 TASKER Keyframe Selector Prompts: Dijkstra selects one segment by comparing boundary frames and prioritizing the most significant UI state transition, including subtle operational changes.Examples include navigation, dialogs, toggles, text input, loading indicators, and panel switching.
  • F.1 TASKER Keyframe Selector Prompts: A* selects one segment by jointly balancing goal proximity with boundary-frame state-change magnitude, without dismissing visually subtle but operationally critical steps.The prompt explicitly considers checkbox toggles, dropdown selections, and typed text as potentially critical actions.
  • F.1 TASKER Keyframe Selector Prompts: QA & Reflect evaluates whether selected frames provide visual continuity sufficient for a user to replicate the task step by step.Confidence 1 denotes severe jumps, 2 minor disconnects, and 3 strong continuity; uncertainty about a major action requires output 1 or 2.
  • F.2 VG-GUI-Bench Evaluation Prompts: VG-GUI-Bench prompts require the model to predict the next action under No Video, Oracle Keyframe, or algorithm-selected-keyframe settings.The no-video baseline provides one current UI screenshot plus the task goal and previous actions as context.
  • F.2 VG-GUI-Bench Evaluation Prompts: In the no-video prompt, the agent understands the goal, locates its workflow position, analyzes visible UI elements, and infers exactly one next action.The prescribed reasoning uses the target screen together with completed previous actions before producing the action call.

G Results on OSWorld

On an OSWorld subset augmented with instructional videos, TASKER-selected frames improved Gemini-3-Flash’s overall task performance, although gains varied by domain. Because OSWorld is not video-guided by design, this evaluation only partially matches TASKER’s target setting.

  • Results on OSWorld: OSWorld is not video-guided by design, so the evaluation only partially matches TASKER’s target setting.Instructional videos were collected for a subset of OSWorld tasks to enable the comparison.
  • Results on OSWorld: TASKER’s performance gains varied across OSWorld domains.The reported overall improvement did not imply uniform gains across domains.
  • Results on OSWorld: TASKER-selected video frames improved Gemini-3-Flash’s overall task success on the OSWorld subset compared with no-video and uniformly sampled frames.The comparison evaluated three conditions: no video, uniform-sampled video frames, and TASKER-selected video frames.
Loading 2606.29445v1…