PaperlayerView exact revision on arXiv

Source-linked AI summary

CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding

Yiduo Jia, Muzhi Zhu, Jinchuan Shi, Hao Zhong, Yuling Xi, Ke Liu, Hao Chen

arXiv:2609.40048v1cs.CV

TL;DR

Ultra-long video grounding must balance long-range evidence search and fine-grained event understanding under a limited visual budget, while existing methods rely largely on predefined policies and tools. CoEvoWhen jointly evolves policies and executable media tools from VLM trajectories into a reusable external skill. Experiments report improved grounding accuracy with lower visual-token cost, and transfer of the skill to long-video QA without task-specific evolution.

  • Problem

    Ultra-long video grounding must locate brief events across extensive timelines while resolving local details and boundaries under a limited visual budget, but existing agentic methods largely rely on predefined observation tools and policies.

    Source: §1 · pp. 1–2

  • Method

    CoEvoWhen jointly refines high-level policies and executable media tools from VLM task trajectories into a reusable external skill without updating model parameters.

    Source: §1 · p. 3

  • Results

    Across five benchmarks and three VLMs, policy–tool coevolution improves grounding accuracy while reducing visual token cost, and on ExtremeWhenBench raises Qwen3.5-27B mIoU by 74.9%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The evolved grounding skill transfers to long-video QA without additional task-specific evolution, and the VLM autonomously uses it without a stronger external planner.

    Source: §1 · p. 3, §5 · p. 10

    Evolution currently uses a limited set of labeled grounding queries from one data source; broader sources, larger scales, and evolution without labeled feedback remain unexplored.

    Source: E LIMITATIONS AND FUTURE WORK · p. 24

Abstract

from arXiv · show

Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely on predefined policies and tool capabilities. Motivated by this, we propose a novel policy-tool coevolution framework that jointly evolves high-level policies and executable media tools from the agentic reasoning trajectories of a VLM, forming a reusable skill without updating model parameters. During evolution, an external skill updater distills transferable task experience in long-video temporal grounding, accordingly refining the orchestration of long-range image-based and fine-grained video-based observations. Alongside these policy updates, the updater employs its coding capabilities to upgrade existing tools or create new ones, adapting the tools to long-video evidence acquisition. Equipped with the evolved skill, the VLM autonomously orchestrates tools under the guidance of the evolved policy, coordinating image and video observations for agentic inference without relying on a separate, stronger planning model. Extensive experiments spanning five benchmarks and three VLMs show that policy-tool coevolution consistently improves temporal grounding accuracy in ultra-long videos while reducing visual token cost at inference, and that the evolved skill yields substantial performance gains on general long-video QA without additional task-specific evolution, demonstrating the effectiveness and generalizability of our framework for long-video understanding.

1 INTRODUCTION

Ultra-long video grounding must balance searching a vast timeline for brief events with locally discerning event details and boundaries under a limited visual budget. CoEvoWhen jointly evolves policies and executable media tools from VLM trajectories; experiments report improved grounding accuracy with lower visual token cost and transfer to long-video QA.

1 INTRODUCTION

  • Ultra-long grounding must find brief target events across vast timelines while resolving local details and temporal boundaries under a limited visual budget.Coarse observation can miss brief events or key details, while repeated fine-grained inspection incurs high visual cost.Source: §1 · p. 1
  • Existing agentic methods leave observation mechanisms and media-processing capabilities largely predefined, despite query-dependent evidence needs.The section asks how task experience can jointly improve tool capabilities and the policies guiding their use for more accurate and efficient evidence acquisition.Source: §1 · p. 2
  • Policies determine what evidence to acquire and how to search, while tools determine the form and granularity of evidence presented to the model.Distant candidate comparison suits compact image observations, whereas action order, state changes, and continuity may require local video observations.Source: §1 · p. 2
  • CoEvoWhen distills grounding trajectories and labeled intervals into a reusable external skill, refining policies and synthesizing code to modify or create media tools.The VLM starts from a minimal skill and autonomously invokes the evolved fixed skill at inference without a separate, stronger planning model.Source: §1 · p. 3
  • On ExtremeWhenBench, evolving Qwen3.5-27B’s base skill raises mIoU by 74.9% while reducing average cumulative visual token cost by 11.4%.Across three ultra-long video grounding benchmarks and two long-video QA benchmarks, coevolution improves grounding accuracy while reducing visual token cost; the evolved skill also improves QA without additional task-specific evolution.Source: §1 · p. 3
  • The framework evolves coordination of complementary image- and video-based observations, and is evaluated across five benchmarks and three VLMs with component ablations.The experiments also examine transfer of the evolved skill to general long-video understanding.Source: §1 · p. 3

2 RELATED WORK

Prior work advances long-video grounding from single-pass prediction toward query-adaptive search and tool use, while often leaving media capabilities predefined. Self-evolving skill work reuses experience, but the cited approaches leave gaps in jointly adapting high-level policies and executable media tools.

2.1 LONG-VIDEO TEMPORAL GROUNDING

  • Long-video grounding progresses from single-pass timestamp prediction to narrowed or recursive search and agentic evidence acquisition, but media-processing capabilities remain largely predefined.Examples include CONE [Hou et al., 2023], SOONet [Pan et al., 2023], ReVisionLLM [Hannan et al., 2025], VideoAgent [Wang et al., 2025c], VideoTree [Wang et al., 2025d], and DVD [Zhang et al., 2025b].Source: §2.1 · pp. 3–4

2.2 SELF-EVOLVING AGENT SKILLS

  • Self-evolving agents reuse experience as guidance, skills, or code, but cited approaches do not jointly update high-level policies and underlying tool implementations.SkillSmith [Wei et al., 2026] restricts tool updates to predefined operations, while META leaves the task-level orchestrator outside evolution.Source: §2.2 · p. 4

3 METHOD

CoEvoWhen iteratively evolves a reusable external skill comprising policies and executable media tools from grounding trajectories, while keeping the VLM parameters fixed. The resulting skill coordinates image and video observations for autonomous, policy-guided inference.

3.1 PROBLEM FORMULATION

  • Temporal grounding localizes all video intervals that semantically correspond to a natural-language query, with the model evaluated on grounding performance and visual token cost.The formulation keeps the VLM parameters fixed while evolving the external skill.Source: §3.1 · p. 4

3.2 POLICY–TOOL COEVOLUTION

  • Across evolution rounds, a frozen VLM executes grounding tasks with its current skill, and an external updater uses the resulting trajectories to update policies and tools.The process begins from a base skill and yields a final skill after K rounds.Source: §3.2 · p. 4, §3.2.2 · p. 5

3.2.1 SKILL REPRESENTATION

  • At round k, the external skill comprises policies for planning and observation orchestration plus executable media tools adapted to diverse observation contexts.Each tool has a capability description, interface specification, and source code; tools may be created, consolidated, or retired.Source: §3.2.1 · p. 4

3.2.2 SKILL UPDATE

  • The updater distills reusable experience into policy refinements and adapts media capabilities by upgrading tools or creating, consolidating, and retiring them.The tool set can also be restructured by retiring unsuitable tools.Source: §3.2.2 · p. 5

3.2.3 COORDINATED IMAGE–VIDEO OBSERVATION

  • Image observations compactly cover extended ranges and compare distant candidates, while video observations preserve local continuity for motion, event order, state changes, and boundaries.The evolved skill coordinates image-based search and candidate refinement with selective video verification.Source: §3.2.3 · p. 5

3.3 POLICY-GUIDED AGENTIC INFERENCE

  • At inference, the evolved policy guides the VLM’s tool choices using the query and accumulated history, and the model submits its final prediction when evidence is sufficient or the round limit is reached.Inference uses the same VLM and does not rely on a separate, stronger planning model.Source: Figure 2 · p. 5, §3.3 · p. 6

4 EXPERIMENTS

Across three ultra-long video grounding benchmarks, the evolved skill improves accuracy, and it reduces visual-token cost while generalizing across VLMs and transferring to long-video QA. Ablations and evolution analysis support jointly adapting policies and media tools, including coordinated image and video observations.

4.1 EXPERIMENTAL SETUP

  • Experiments use three grounding benchmarks and two long-video QA benchmarks, comparing base and evolved skills under identical protocols and inference settings.The main evolution uses Qwen3.5-27B and an external updater; one pass over the evolution set updates policies and tools after batches of four queries, with VLM weights frozen.Source: §4.1 · p. 6

4.2.1 ULTRA-LONG VIDEO TEMPORAL GROUNDING

  • The evolved skill achieves the best reported results across grounding metrics, raising VUE-LVTR held-out IoU AUC from 0.4107 to 0.5137 and improving ExtremeWhenBench and CoMET-Bench results over the base skill.ExtremeWhenBench mIoU and Recall@0.5 improve by 74.9% and 88.4%; CoMET-Bench mIoU gains 0.0280 and Rejection-F1 gains 10.93 points.Source: Table 1 · p. 7, §4.2.1 · p. 7

4.2.2 INFERENCE EFFICIENCY

  • Average visual tokens per query fall from 202.6k to 141.9k (30.0% fewer) on VUE-LVTR, with reductions of 11.4% on ExtremeWhenBench and 18.9% on CoMET-Bench.On ExtremeWhenBench, image-observation tokens rise from 179.3k to 197.5k while video-observation tokens fall from 47.8k to 3.7k; model calls also decrease across all three benchmarks.Source: Table 2 · p. 7, §4.2.2 · pp. 7–8

4.2.3 CROSS-VLM GENERALIZATION AND CROSS-TASK TRANSFER

  • Coevolution improves ExtremeWhenBench mIoU by 99.2% on Qwen3.5-9B and 24.7% on Qwen3.6-27B, while direct long-video QA transfer raises Qwen3.5-27B accuracy by 8.65 points on LVBench and 7.98 on LSDBench.The grounding skill is applied to QA without additional task-specific evolution.Source: Table 3 · p. 8, §4.2.3 · p. 8

4.3.3 ANALYSIS OF SKILL EVOLUTION

  • During evolution, grounding accuracy more consistently rises as visual-token cost falls, while trajectory-exposed omissions, boundary errors, and dynamic ambiguities drive coordinated policy and tool adjustments.Image tools with candidate-retention and local-refinement policies target evidence coverage and boundary judgment; video observations target action verification, order, and continuity.Source: Figure 4 · p. 9, §4.3.3 · p. 9, Figure 5 · p. 9

4.3.1 ABLATION ON POLICY–TOOL COEVOLUTION

  • Joint policy–tool evolution exceeds policy-only and tool-only IoU AUC by 0.0243 and 0.0735, respectively, while reducing visual-token cost by 40.6% and 25.5%.Both variants improve accuracy over the base skill, but freezing policies during evolution causes a larger accuracy loss than freezing tools.Source: Figure 3 · p. 8, §4.3.1 · p. 9

4.3.2 ABLATION ON IMAGE–VIDEO COORDINATION

  • The evolved image+video skill outperforms evolved image-only and video-only skills across grounding metrics while using fewer visual tokens.The ablation restricts observations to one modality during evolution and evaluation; the table compares image+video gains and token reductions against those single-modality skills.Source: Table 4 · p. 8, §4.3.2 · p. 9

5 CONCLUSION

The conclusion presents CoEvoWhen as a reusable external skill that jointly evolves policies and executable tools from VLM trajectories, improving grounding accuracy while reducing visual-token cost. The appendix describes the evolution queries, base-skill structure, and settings used for evaluation and transfer.

5 CONCLUSION

  • CoEvoWhen jointly evolves high-level policies and executable media tools from agentic trajectories into a reusable skill, and experiments report grounding gains, lower visual-token cost, cross-VLM gains, and long-video QA transfer.Inference uses the evolved skill with autonomous tool orchestration, without a stronger external planner or additional task-specific evolution for QA.Source: §5 · p. 10

A APPENDIX OVERVIEW

  • The appendix overview directs readers to implementation details, additional results, case studies, limitations and future work, and prompts.It identifies sections B through F and their stated topics.Source: A APPENDIX OVERVIEW · p. 14

B.1.1 EVOLUTION SET CONSTRUCTION

  • The 100-query evolution set comes from distinct videos and includes long and ultra-long videos, with selection spanning sparse targets, short intervals, and varied search and localization needs.Eighty queries have sparse target events, and 88 contain at least one target interval no longer than 10 seconds.Source: §B.1.1 · p. 14, Table 5 · p. 14

B.1.2 BASE SKILL

  • The base skill contains policy documents, tool interfaces and source code, but its initial policies provide only basic grounding and observation guidance rather than advanced search or modality orchestration.Its six tools include media preparation, image and video observation, and final answer submission; local tools prepare inputs without calling additional visual models.Source: §B.1.2 · p. 15, Table 6 · p. 15

B.1.3 EVOLUTION SETTINGS

  • The main evolution uses Qwen3.5-27B and Codex, updating policies and tools after each four-query batch in a single pass over 100 queries while keeping model parameters fixed.Separate skills are evolved for cross-VLM experiments; the grounding skill is applied directly to long-video QA without further task-specific evolution.Source: §B.1.3 · p. 15

B.2.1 DATA SPLITS

The evaluation splits use 307 held-out VUE-LVTR queries after 100 queries are selected for evolution, the full ExtremeWhenBench test set, and a duration-filtered CoMET-Bench set.

B.2.1 DATA SPLITS

  • VUE-LVTR contains 407 queries, with 100 used for evolution and 307 disjoint queries held out; evaluation uses all 2,273 ExtremeWhenBench queries and 1,599 CoMET-Bench queries with videos at least 30 minutes long.The VUE-LVTR queries are drawn from VUE-TR and VUE-TR-V2; the CoMET-Bench evaluation retains target-present and target-absent cases.Source: §B.2.1 · p. 15, Table 7 · p. 16

B.2.2 EVALUATION METRICS

This section defines benchmark-specific grounding and event metrics, visual-token accounting, and shared inference settings for comparing fixed base and evolved skills.

B.2.2 EVALUATION METRICS

  • VUE-LVTR rounds predicted boundaries to integer seconds, merges overlapping or adjacent intervals, and uses precision, recall, IoU, and their threshold-range AUCs.IoU AUC integrates hit rates over thresholds from 0 to 1 in steps of 0.01.Source: §B.2.2 · p. 16
  • ExtremeWhenBench scores single-interval predictions by temporal IoU, including invalid predictions as zero, and reports mIoU across five query categories.The category metrics are Action, Environment, Object, Reaction, and Scene.Source: §B.2.2 · p. 16
  • CoMET-Bench evaluates multi-event grounding, event counts, and absent-target queries using grounding scores, counting metrics, rejection behavior, and positive-query coverage.Counting uses MAE, off-by-one accuracy, and Pearson correlation; rejection metrics include false positive rate and Rejection-F1.Source: §B.2.2 · pp. 16–17
  • Average cumulative visual cost counts image and video tokens received across model calls, counting tokens again when media re-enters interaction history.Generated media not passed to the model incurs no cost, and all queries contribute to the average.Source: §B.2.2 · p. 17

B.2.3 INFERENCE SETTINGS

  • Base and evolved skills are fixed during evaluation and share prompts, decoding, interaction budgets, and scoring; experiments use visual inputs only.Transferred long-video QA uses grounding-evolved policies and tools without updating the skill on QA data.Source: §B.2.3 · p. 18, Table 8 · p. 18

B.2.4 BASELINE EVALUATION PROTOCOLS

The baseline protocol compares seven VLM-based and agent-based systems across the three ultra-long video grounding benchmarks.

B.2.4 BASELINE EVALUATION PROTOCOLS

  • Seven baselines span open-source and closed-source VLMs with different frame-input configurations, plus VideoMind’s collaborative reasoning and EvoGround’s Proposer–Solver feedback.The VLM frame configurations range from 128 to 768 frames.Source: §B.2.4 · p. 18

C.1 RESULTS ON THE EVOLUTION SET

Evaluation on the 100 evolution queries shows gains across three VLMs, while held-out token costs fall across three benchmarks for Qwen3.6-27B.

C.1 RESULTS ON THE EVOLUTION SET

  • IoU AUC gains on the 100 evolution queries are 0.0882 for Qwen3.5-9B, 0.0651 for Qwen3.5-27B, and 0.0523 for Qwen3.6-27B.These fixed-skill results characterize fit to the evolution data.Source: §C.1 · p. 18, Table 9 · p. 19

C.2 VISUAL TOKEN COST ON QWEN3.6-27B

  • Visual token cost decreases by 42.2% on VUE-LVTR, 28.1% on ExtremeWhenBench, and 15.8% on CoMET-Bench for Qwen3.6-27B.On CoMET-Bench, image tokens rise from 83.60k to 100.19k while video tokens fall from 55.36k to 16.82k.Source: §C.2 · pp. 18–19, Table 10 · p. 19

C.3 RESULTS ACROSS MULTIPLE DIMENSIONS

The reported gains extend across query categories, event counting, rejection of absent targets, and multi-event grounding, with variation by model and query type.

C.3 RESULTS ACROSS MULTIPLE DIMENSIONS

  • Qwen3.5-9B and Qwen3.5-27B improve in all five ExtremeWhenBench categories, while Qwen3.6-27B improves in four and slightly declines on Environment.The distribution of gains varies with both model and query type.Source: §C.3 · p. 19, Table 11 · p. 20
  • On CoMET-Bench, counting MAE decreases and OBO and Pearson correlation increase for all three VLMs after evolution.The authors report gains in event-count accuracy.Source: §C.3 · p. 19, Table 12 · p. 20
  • All three VLMs improve Rejection-F1 and positive-query coverage, while Qwen3.5-9B and Qwen3.5-27B also lower false positive rate.Source: §C.3 · p. 19, Table 13 · p. 20
  • Qwen3.5-9B and Qwen3.5-27B improve across all event-count groups, and Qwen3.6-27B improves in four of five, but higher event counts retain lower F1@0.5.Precisely detecting and localizing multiple target events remains challenging.Source: §C.3 · p. 19, Table 14 · p. 21

D CASE STUDY

Across two ultra-long-video cases, the evolved skill combined compact global search with targeted local verification or boundary refinement, producing more precise intervals at lower visual token cost. The cases illustrate how observation choices adapt to whether the query requires motion verification or scene-boundary localization.

D CASE STUDY

  • The case analysis used two ExtremeWhenBench queries with the same Qwen3.5-27B, and both skills remained fixed during evaluation.Visual token costs followed the definitions in Appendix B.2.2.Source: D CASE STUDY · p. 21
  • The evolved skill localized candidates before resolving remaining uncertainty, using video to verify action progression and images to determine scene boundaries.In the base trajectories, additional local evidence remained confined to an incorrect candidate; evolved tools provided compact temporal coverage and explicit temporal correspondence.Source: D CASE STUDY · p. 21
  • 51.27% lower visual token cost accompanied an IoU increase from 0 to 1, as the evolved skill localized the fries-scooping action to the ground-truth interval [570, 582] s.The base skill repeatedly inspected a visually similar chute transfer; the evolved skill used timestamped contact sheets, dense frames around 550–590 s, and one 15-second clip.Source: D CASE STUDY · p. 21, Figure 6 · p. 22
  • 45.78% lower visual token cost accompanied an IoU increase from 0 to 1, as contact-sheet search and boundary frames localized the brief scene to [5703, 5714] s without video observation.The base skill missed the brief target during sparse sampling and selected a visually similar scene; the evolved skill packed 192 frames into eight contact sheets.Source: D CASE STUDY · p. 21, Figure 7 · p. 23

E LIMITATIONS AND FUTURE WORK

The authors limit current coevolution to a small set of labeled temporal-grounding queries from one data source and identify broader data, larger evolution scales, unlabeled feedback, delegated observation, and omni-modal settings as future work. The accompanying prompts specify task outputs and reusable-skill update procedures.

E LIMITATIONS AND FUTURE WORK

  • Current coevolution uses a limited set of temporal-grounding queries from one data source with labeled feedback; broader data, larger evolution scales, unlabeled feedback, subagent coordination, and omni-modal settings remain future work.The current framework has the VLM main agent perform visual observations directly.Source: E LIMITATIONS AND FUTURE WORK · p. 24

System Prompt

  • The system prompt frames the solver as a video-task agent using an active evolvable skill and an optional task-specific adapter.The listed skill components are SKILL.md, tool-selection.md, tool-notes.md, and policy-notes.md.Source: System Prompt · p. 24

VUE-LVTR Prompt

  • The VUE-LVTR prompt requires a JSON array of matching source-video intervals, with each interval expressed as start and end seconds.Source: VUE-LVTR Prompt · p. 25

ExtremeWhenBench Prompt

CoMET-Bench Prompt

  • The CoMET-Bench prompt requests a JSON array of event intervals and specifies an empty array when the event does not occur.Source: CoMET-Bench Prompt · p. 25

Long-Video QA Prompt

  • The long-video QA prompt reuses the active skill for evidence localization and verification, then requires exactly one option letter; temporal-output conventions do not apply.Source: Long-Video QA Prompt · p. 25

Codex Updater Prompt

  • The updater must avoid copying task-specific scene facts or predictions into the skill and may edit only the candidate skill files.Its instructions prohibit external models or APIs, audio, speech recognition, and transcripts.Source: Codex Updater Prompt · p. 26
  • The updater analyzes trajectories and outcomes to revise reusable evidence workflows, tool contracts, selection guidance, and deterministic local media-preparation tools.Updates can cover sampling, clipping, timestamp labeling, compact visual summaries, motion- or scene-aware preparation, validation, and tool consolidation or deletion.Source: Codex Updater Prompt · p. 26
Loading 2609.40048v1…