Source-linked AI summary
Can Agents Win the Video Browser Showdown?
Bastian Jäckl, Zuzana Vopálková, Daniel A. Keim, Jakub Lokoč
TL;DR
Interactive video search requires users to translate intent into queries and iteratively assess and refine results, while it remains unclear how well agents can autonomously manage diverse tasks. This paper evaluates a multi-agent system that operates a video retrieval backend from specified search intents. On historical VBS tasks, agents performed comparably to strong expert-operated systems in several settings, with important scope limits in visual access and human comparison.
Problem
It remains unclear whether agents can autonomously manage diverse interactive video-search tasks or how their performance compares with strong expert-operated systems.
Source: §1 · p. 2Method
A fully autonomous multi-agent system uses a retrieval backend to execute interactive search from specified intents, with visual task material represented to agents as text.
Source: §1 · p. 2, §3.1 · pp. 4–5Results
Across 81 historical VBS tasks, agents performed comparably to strong historical expert-operated systems in several t-KIS, AVS, and VQA settings, but remained limited on homogeneous visual collections and fine-grained information extraction.
Source: §5 · p. 13Takeaways & Limitations
The results suggest a division of labor in which users specify intent, agents execute and adapt retrieval, and humans provide relevance feedback and final verification, particularly for fine-grained judgments.
Source: §5 · p. 13The historical expert comparison is indicative rather than controlled, and visual-task agents search from textualized intent without human-equivalent perceptual access to the original material.
Source: §3.1 · p. 5, §4.1 · p. 8
Abstract
from arXiv · showhide
Searching large video collections is typically an interactive process in which users play two roles. First, they hold the search intent: the underlying goal that determines what content they seek and why. Second, users must operationalize this intent through an iterative search loop. Users translate their intent into queries, browse the retrieved candidates, and refine their queries based on the results. In this paper, we investigate the capabilities of modern Vision Language Models (VLM) and agentic approaches to reach search goals interactively and fully autonomously. Specifically, we study whether a provided initial specification of a search goal might be sufficient to solve traditionally interactive search tasks with an agentic system. Provided that the involved VLMs are not aware of the whole large video dataset in advance, the key challenge lies in the effective combination of an existing interactive video search system and a smart VLM agent controlling the system. While the search system provides indexing and efficient querying, the VLM-based agents analyze top-ranked items and make decisions about next actions. Our results show that modern agents can autonomously operate interactive video retrieval systems to solve many search tasks from an initial intent description, achieving performance competitive with strong historical expert-operated systems in several settings.
1 Introduction
Interactive video retrieval depends on users translating intent into queries and iteratively assessing and refining results, a process that can be cognitively demanding and expert-dependent. This paper tests whether autonomous agents can execute that operational role from a fixed, well-specified intent and establishes a baseline against historical expert runs.
1 Introduction
- Interactive retrieval can be cognitively demanding, time-consuming, error-prone, and reliant on expert knowledge, making user experience relevant to retrieval success.Systems offer textual, visual, and temporal queries, relevance feedback, filtering, keyframe layouts, and video players.Source: §1 · p. 1
- Agents can autonomously manage a broader range of interactive search capabilities across diverse video tasks, a question not resolved by prior ad-hoc retrieval work.The paper also notes that agentic video search had not been systematically evaluated against strong expert-operated systems across such tasks.Source: §1 · p. 2
- The study isolates search execution from intent specification, asking whether agents can formulate queries, assess results, refine strategies, and determine when candidates satisfy a fixed intent.The authors characterize this controlled setting as an optimistic estimate of real-world performance because users may express needs incompletely or ambiguously.Source: §1 · p. 2
- The system combines a fully autonomous multi-agent framework with the PraK backend to operate across four VBS task types.It is evaluated on three years of historical tasks and contextualized against corresponding expert performance.Source: §1 · p. 2
- The paper contributes analyses of agent behavior and failure modes, open-source code, and a visual interface for inspecting search traces.The contribution passage also identifies the code as open-source.Source: §1 · p. 3
2 Related Work
Interactive retrieval systems combine indexed collections, multimodal retrieval and refinement, and interfaces for users to inspect candidates. Prior agentic approaches retain user involvement, use predefined workflows, or focus on textual ad-hoc retrieval; this work evaluates fully autonomous closed-loop search across heterogeneous tasks.
2 Related Work
- Interactive video retrieval systems combine indexed video collections, multimodal retrieval and refinement operations, and interfaces for query formulation and candidate browsing.Powerful multimodal embeddings support semantic retrieval from natural-language queries, but complex or visually similar candidates still require inspection and iterative refinement.Source: §2 · p. 3
- Prior agentic video-search approaches either retain user involvement, rely on predefined workflows, or focus primarily on textual ad-hoc retrieval.VRAgent adapts multimodal instructions using top-result confidence, while Wu et al. [32] assess candidates and choose whether to continue inspection or reformulate queries.Source: §2 · p. 3
- This work extends closed-loop agentic search to four heterogeneous VBS task types under official time constraints and compares performance with historical expert-operated systems.Source: §2 · p. 3
3 Agentic Video Search: Benchmark and System
The paper establishes an agentic baseline on VBS, whose controlled tasks, time limits, and archived expert results support standardized evaluation. Its autonomous system represents tasks for agents, then uses a planner, shared state, retrieval backend, and specialized inspection and answering modules to execute a closed-loop search.
3 Agentic Video Search: Benchmark and System
- VBS provides controlled search intents, diverse task types, official time limits, and archived expert results for standardized evaluation of autonomous search.The agents formulate queries, inspect candidates, refine searches, and return results under the benchmark time limits without human intervention.Source: §3.1 · p. 4
3.1 VBS Setting and Tasks
- The Task Encoder maps task type, material, and time limit into the agent specification, preserving textual descriptions while encoding visual material as fixed text.The study uses both automatically generated VLM descriptions and human descriptions to distinguish encoding limitations from limitations of the subsequent search process; VQA also appends the question.Source: §3.1 · p. 4
- Visual-task evaluation uses textual descriptions instead of original videos or frames, so agent search does not have perceptual access equivalent to human participants.The original visual material is unavailable to agents during the timed search; the paper notes that no task representation yields a strictly equivalent comparison.Source: §3.1 · pp. 4–5
3.2 Agentic Search System
- A planner coordinates retrieval and inspection through persistent shared state that accumulates actions, candidate assessments, video evidence, and benchmark feedback.The planner reads this state to select actions, while modules add observations, enabling evidence accumulation and avoiding repeated unsuccessful actions.Source: §3.2 · pp. 5–6
- PraK supplies multimodal semantic, temporal, and relevance-feedback retrieval, with validated actions constrained by the current search state.The validation harness enforces action preconditions, including that feedback uses the latest result set and identifier-based actions refer only to items already in state.Source: §3.2 · p. 6
- The browser assesses keyframes and surrounding scenes, while the inspector answers targeted questions and VQA strategies search relevant, exhaustive, or chronological scene sequences.Shortly before the deadline, a fallback estimator submits the most likely answer if the retrieval loop has not found a definitive one.Source: §3.2 · pp. 6–7
4 Experiments and Results
Across VBS tasks, agents achieved task-dependent performance relative to expert references. Their behavioral patterns show iterative retrieval, while visual-task performance was constrained by textualized task descriptions.
4.1 Implementation and Experimental Settings
- The evaluation covered 81 tasks from the 2024–2026 VBS editions, but human comparisons are indicative because participants used different systems and could restart searches.The agents used the PraK backend operated by 2026 PraK experts, making those comparisons more controlled than comparisons with other systems or earlier configurations.Source: §4.1 · pp. 7–8
4.2 Performance of Agentic Search
- Each agent run solved 9 of 11 t-KIS tasks, and on 8 of 9 tasks also solved by experts, at least one agent run beat the fastest expert time.The figure also shows solve time, wrong answers, and first-query target-rank distribution across retrieval depths.Source: §4.2 · p. 8, Figure 2 · p. 9
- In successful runs, 46% of targets ranked outside the first 100 results initially, yet many were found quickly through subsequent query reformulation or retrieval-strategy changes.Sequentially browsing deeper into the initial ranking would not have reached these targets within the observed completion times.Source: §4.2 · p. 9
- Agents matched or exceeded the best expert run in 8 of 13 AVS tasks; falsification rejected 56 of 169 incorrect submissions (33.1%) and 27 of 552 relevant submissions (4.9%).The AVS figure compares agent and human performance; the falsification agent searched five-second candidate clips for contradictory evidence.Source: §4.2 · pp. 9–10, Figure 3 · p. 10
- Agents approached expert performance on V3C v-KIS but fell substantially behind on MVK and solved none of the LHE tasks.In 146 of 154 unsuccessful MVK and LHE runs (94.8%), the target never appeared among candidate results; the authors identify fixed textual task representation as a likely contributor.Source: §4.2 · p. 10, Figure 4 · p. 11
- VLM-description VQA runs solved 11–12 of 16 tasks, compared with 12–14 for experts; four tasks remained unsolved across all VLM runs.For those four, the correct video and answer-containing scene were retrieved, but the requested information was not extracted; all required fine-grained object recognition.Source: §4.2 · p. 11, Figure 5 · p. 11
4.3 Behavioral Analysis
- The planner’s next action was associated with browser feedback: after no relevant keyframes, it reformulated queries in 78.2% of cases, versus 18.4% for IVQ and 3.4% for Bayes.The analysis characterizes responsiveness to evidence rather than optimality; relevant-result counts alone did not explain the choice among refinement strategies.Source: §4.3 · p. 12
- After excluding targets retrieved by the first semantic search, 30.9% were reached through other retrieval strategies, predominantly IVQ (26.6%).Across tasks, semantic search was the most frequent action, and action sequences commonly alternated between collection and single-video search.Source: §4.3 · p. 12
5 Conclusion
The autonomous system established an end-to-end baseline across historical VBS tasks, with performance comparable to strong historical expert-operated systems in several settings. The results also point to a division of labor in which agents adapt retrieval operations while humans specify intent and provide feedback and verification.
5 Conclusion
- Across 81 historical VBS tasks, agents achieved performance comparable to strong historical expert-operated systems in several t-KIS, AVS, and VQA settings.The evaluation established an end-to-end baseline for autonomous interactive video retrieval.Source: §5 · p. 13
- Performance remained limited for homogeneous visual collections and fine-grained information extraction.Source: §5 · p. 13
- Agents formulated queries, used different retrieval tools, assessed candidates, and adapted their strategy using intermediate feedback.They rapidly recovered targets ranked beyond the inspectable range after the initial query, underscoring the iterative retrieval loop.Source: §5 · p. 13
- The authors suggest users specify intent while agents execute and adapt retrieval, with humans providing relevance feedback and final verification, especially for fine-grained judgments.They characterize the results as an initial agentic baseline rather than a controlled human–agent comparison.Source: §5 · p. 13