PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding

Weixin Xu, Zhenyu Yang, Bing Wang, Shengsheng Qian, Changsheng Xu

arXiv:2609.15606v1cs.CVcs.AI

TL;DR

Long videos exceed MLLMs’ visual context windows, and fixed sampling or sparse first-pass zooming can miss decisive evidence. VideoScout treats understanding as sequential evidence acquisition, exploring turn by turn with adaptive pacing and a two-stage training pipeline. It achieves the best open-source results across both benchmark categories reported, averaging 48.0 on long video understanding and reasoning and 45.0 on cross-domain video understanding.

  • Problem

    Long videos exceed MLLMs’ visual context windows, while fixed frame selection or sparse first-pass scans can miss decisive evidence.

    Source: §I · p. 1

  • Method

    VideoScout explores videos turn by turn using adaptive speed, textual memory, and rewatch actions, trained with turn-level SFT and trajectory-level reinforcement learning.

    Source: §I · p. 1, A. Overview · p. 4

  • Results

    VideoScout averages 48.0 on long video understanding and reasoning and 45.0 on cross-domain video understanding, the best performance among open-source methods on both categories.

    Source: B. Main Results · p. 8

  • Takeaways & Limitations

    The results support active exploration with adaptive reasoning pacing as an effective approach for long-video understanding across the reported benchmark categories.

    Source: B. Main Results · p. 8, §V · p. 14

    Rewatching is limited to the most recent fast-forward window, so the agent may be unable to revisit much earlier chunks later in a long video.

    Source: G. Limitations · p. 13

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have achieved remarkable progress on short video understanding yet remain limited on long videos due to the limited visual context window. Prevailing approaches rely on uniform frame sampling or recent coarse-to-fine agentic zooming, both of which struggle to localize sparse, decisive evidence in sufficiently long videos. We formulate long video understanding as a \textbf{Sequential Evidence Acquisition (SEA)} problem, in which an agent reads the video turn by turn along the temporal axis, deciding at each turn how fast to watch, what evidence to retain, when to revisit uncertain segments, and when to stop and answer. Inspired by this view, we propose \textbf{VideoScout}, a multi-turn reasoning agent that instantiates the SEA paradigm through adaptive reasoning pacing. Specifically, by dynamically controlling the viewing pace, VideoScout enables efficient traversal of long videos within a bounded visual context window, allowing the agent to access more video content while balancing content analysis depth with reading efficiency. To train VideoScout, we construct VideoScout-66K, a set of over 66K high-quality exploration turns from 10K answer-verified trajectories, and adopt a two-stage pipeline: cold-start supervised fine-tuning teaches the agent per-turn output format, while the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm performs trajectory-level reinforcement learning with a composite reward that jointly considers answer accuracy, output format compliance, and the temporal alignment between the agent's viewing progress and the teacher's answer timing measured by intersection-over-union (IoU). Extensive experiments on long video understanding and reasoning benchmarks demonstrate that our 7B model achieves strong performance compared with existing trained 7B agentic models.

I. INTRODUCTION

Long-video understanding must select useful evidence from videos that exceed the visual context window. VideoScout frames this as sequential evidence acquisition and uses turn-by-turn exploration, adaptive pacing, and two-stage training to address it.

I. INTRODUCTION

  • Long videos exceed current 7B MLLMs’ visual context windows, making the selected frames decisive for which evidence is available to answer a query.A two-hour film at 24 fps contains over 170,000 frames; even at 1 fps, it exceeds the context window by more than an order of magnitude.Source: §I · p. 1
  • VideoScout couples cross-turn textual notes and rewatching to gather sparse evidence iteratively, while adaptive speed concentrates bounded visual capacity on segments worth examining.It can fast-forward through uninformative passages at 2× or 4× and slow to 1× on segments worth examining.Source: §I · pp. 1–2
  • Sequential Evidence Acquisition (SEA) formulates long-video question answering as turn-by-turn temporal reading, with decisions about pace, retained evidence, revisits, and when to answer.The formulation recasts encoding a fixed video as allocating limited observations over time.Source: §I · p. 2
  • VideoScout-66K contains over 66K verified exploration turns from 10K answer-verified trajectories, and training combines turn-level cold-start SFT with trajectory-level DAPO.DAPO’s composite reward considers answer accuracy, output-format compliance, and temporal alignment with teacher viewing behavior measured by IoU.Source: §I · pp. 2–3

II. RELATED WORK

Prior long-video methods either commit to frames before reasoning or rely on an initial sparse overview; the paper instead develops a trained agent that explores across turns and optimizes its full trajectory.

A. Long Video Understanding

B. Tool-Augmented Video Agents with Reinforcement Learning

  • Single-turn video RL methods reason over passively provided inputs, while coarse-to-fine agents remain anchored to an initial global scan that becomes less reliable as video duration increases.VideoScout instead formulates long-video understanding as multi-turn reasoning under bounded visual context and applies a composite DAPO reward.Source: B. Tool-Augmented Video Agents with… · p. 3

III. METHODOLOGY

  • VideoScout divides a long video into contiguous chunks and processes one current clip with inherited textual memory at each turn until the agent answers.Its per-turn output includes a reasoning trace, optional memory update, and one action: speed, rewatch, or answer.Source: A. Overview · p. 3

A. Overview

  • Rewatching examines a chunk in the current multi-chunk window at 1× before forward progress resumes, while bounded clip and memory sizes keep per-turn inference cost constant in video length.Cold-start SFT trains individual turns; RL optimizes complete multi-turn trajectories, aligning optimization with the exploration process.Source: A. Overview · p. 4

B. Data Generation Pipeline

The data pipeline creates answer-verified exploration trajectories, filters them for quality, and supplies both turn-level supervision and a rebalanced prompt pool for reinforcement learning.

B. Data Generation Pipeline

  • Gemini 3 Flash generates multi-turn teacher trajectories using the system prompt, action space, Think-Act format, and memory mechanism without access to the ground-truth answer.Each query is paired with its source video and ground-truth answer.Source: B. Data Generation Pipeline · pp. 4–5
  • Incorrect teacher trajectories are resampled up to eight times and discarded if no correct answer is produced; additional validation checks reasoning consistency, answer-tag agreement, truncation, and meaningful traces.Only trajectories with correct final answers are retained.Source: B. Data Generation Pipeline · p. 5
  • The pipeline yields approximately 10K verified trajectories comprising over 66K individual turns; these trajectories are used for supervised fine-tuning, and approximately 49% of samples exceed 30 minutes.Source: B. Data Generation Pipeline · p. 5
  • The roughly 4K-prompt RL pool combines high-quality validated queries with additional Video-R1 samples and rebalances video durations, including 33% over 30 minutes.Its duration distribution is more uniform than the SFT set.Source: B. Data Generation Pipeline · p. 5

C. Cold-Start Supervised Fine-Tuning

Cold-start supervised fine-tuning treats each trajectory turn as an independent example, teaching VideoScout its structured output and per-turn action behavior.

C. Cold-Start Supervised Fine-Tuning

D. Trajectory-Level Reinforcement Learning

VideoScout is optimized with trajectory-level DAPO over complete multi-turn rollouts, using a composite reward for answer quality, output validity, and timing alignment. Dynamic sampling and decoupled clipping shape updates toward informative exploration.

1) Multi-turn rollout

  • DAPO optimizes complete multi-turn rollouts sampled from roughly 4K prompts, using a group-relative advantage and a clipped surrogate objective over output tokens.Dynamic sampling excludes prompt groups whose trajectories are all correct or all incorrect, then samples until the batch contains informative prompts.Source: D. Trajectory-Level Reinforcement… · pp. 5–6
  • Separate upper and lower clipping thresholds allow more aggressive exploration in the trajectory-level policy update.The objective applies a per-token clipped surrogate across each trajectory’s output turns.Source: D. Trajectory-Level Reinforcement… · p. 6

2) Composite reward

  • The terminal reward combines answer accuracy, output validity, and alignment between the agent’s viewing progress and the teacher’s answer timing.Timing reward is awarded only when the answer is correct, preventing timing alignment from replacing accuracy.Source: D. Trajectory-Level Reinforcement… · pp. 6–7

A. Experimental Setup

VideoScout is initialized from Qwen2.5-VL-7B-Instruct and trained in two stages, with separate supervised fine-tuning and reinforcement-learning settings. Evaluation spans long-video reasoning and cross-domain understanding, against proprietary and open-source single-pass and agentic baselines.

1) Implementation Details

  • Cold-start SFT uses over 66K exploration turns from 10K answer-verified trajectories for 3 epochs on 8 NVIDIA H800 GPUs, followed by one RL epoch on the same hardware.SFT uses a 1.5×10^-5 learning rate and batch size 16; RL uses a 5×10^-6 learning rate and group size 8.Source: A. Experimental Setup · p. 7

2) Benchmarks

  • The nine benchmarks cover long-video understanding and reasoning as well as cross-domain video understanding across diverse knowledge domains.The long-video group includes perception, memory, temporal and causal reasoning, and multi-step reasoning; the cross-domain group includes multidisciplinary, scientific, commonsense, and visual-spatial tasks.Source: A. Experimental Setup · p. 7

3) Baseline Methods

  • Baselines include proprietary single-pass MLLMs and open-source single-pass models and agentic frameworks, plus an untrained agentic Qwen2.5-VL-7B variant.The untrained variant uses the same multi-turn exploration framework as VideoScout but not its two-stage training.Source: A. Experimental Setup · p. 7

B. Main Results

VideoScout leads the reported open-source averages in both long-video understanding and cross-domain understanding. Its results also compare favorably with base and untrained agentic variants, while its observation and reasoning effort varies with video length.

Main Results

  • 48.0 on long-video understanding and reasoning and 45.0 on cross-domain video understanding are the highest open-source averages reported.The comparisons use frame budgets for other methods that do not significantly exceed VideoScout's.Source: B. Main Results · p. 8, Table III · p. 8, Table IV · p. 9

Long Video Understanding and Reasoning

  • 45.1 on LVBench, 35.4 on MINERVA, and 51.0 on MLVU-test are the best open-source results, while VideoScout scores 60.4 on Video-MME.VideoScout ranks second-best on Video-MME among the compared methods.Source: B. Main Results · p. 8, Table III · p. 8

Comparison with Base and Agentic Variants

  • VideoScout improves the long-video average by 6.2 points over Qwen2.5-VL-7B and by 14.9 over its untrained agentic variant.It also leads Video-MTR-7B and VideoTemp-o3-7B on every reported metric.Source: B. Main Results · p. 8

Adaptive Observation

  • VideoScout spends more frames and reasoning turns on longer benchmarks such as LVBench and fewer on shorter ones, rather than following a fixed observation schedule.The authors report this allocation in Table V.Source: B. Main Results · p. 8, Table V · p. 9

Cross-Domain Video Understanding

  • 45.0 is VideoScout's highest open-source average on cross-domain benchmarks, with first-place scores of 54.6 on VideoMMMU, 28.9 on SciVideoBench, and 40.8 on WorldSense.It also scores 65.9 on VideoMMMU perception, second only to Open-o3-Video-7B, and 35.0 on VSI-Bench and 65.7 on MMVU-mc.Source: B. Main Results · p. 8, Table IV · p. 9

Cross-Domain Comparison

  • On cross-domain benchmarks, VideoScout gains 7.2 points over Qwen2.5-VL-7B and 10.7 over its untrained agentic variant.The authors attribute the gains to two-stage training rather than the agentic framework alone.Source: B. Main Results · p. 8

C. Ablation Study

The ablations test VideoScout's training stages, reward components, and adaptive exploration choices. They report benefits from two-stage training and from adapting chunk duration, speed, frame budget, and memory mechanisms.

Two-Stage Training Pipeline

  • SFT+RL with DAPO reaches 45.1 on LVBench, 54.6 on VideoMMMU, 51.0 on MLVU, and 35.4 on MINERVA, outperforming SFT alone on all four.RL without SFT initialization collapses across the benchmarks; GRPO on top of SFT improves only marginally on LVBench and degrades on VideoMMMU and MINERVA.Source: C. Ablation Study · p. 9

Composite Reward

  • Removing the format reward lowers LVBench from 45.1 to 42.3, while removing the timing reward consistently reduces performance and leaves total reward nearly unchanged during RL.The timing-reward curves show the training trend; the authors describe the three reward signals as complementary.Source: C. Ablation Study · p. 10, Table VII · p. 10, Figure 6 · p. 10

Frame Budget

  • The default frame budget scores 45.1 on LVBench, versus 44.3 with more frames; fewer frames and uniform high resolution also fail to improve consistently across benchmarks.The more-frames variant is retrained with RL under its own configuration; the authors report that extra frames can add redundant content while fewer frames lose fine-grained cues.Source: C. Ablation Study · p. 10, Table VIII · p. 10

Chunk Duration

  • Adaptive chunk duration outperforms fixed 10- and 30-second chunks; the 10-second setting drops LVBench from 45.1 to 42.2.The authors link fixed granularity to a mismatch with varying video lengths: short chunks add turns, while long chunks reduce temporal resolution.Source: Table VIII · p. 10, C. Ablation Study · p. 11

Speed Policy

  • The mixed-speed policy outperforms both single-speed alternatives: slow-only scores 43.6 on LVBench, while fast-only scores 41.7.The authors report that slow-only spends reasoning on irrelevant content, whereas fast-only can skip fine-grained details.Source: Table VIII · p. 10, C. Ablation Study · p. 11

Core Mechanism

  • Removing Notes drops MLVU from 51.0 to 42.0, and removing rewatch lowers LVBench from 45.1 to 42.5.Notes aggregate query-relevant observations across turns; rewatch lets the agent recover details missed while fast-forwarding.Source: Table VIII · p. 10, C. Ablation Study · p. 11

D. Efficiency Analysis

VideoScout's inference time varies with benchmark demands, and its reported accuracy is highest on all four evaluated benchmarks. It uses less time for short-range comprehension and more for long-video reasoning.

Evaluation Setup

  • Measurements use average inference time per sample and accuracy on one NVIDIA H800 GPU with batch size one.Each sample is processed independently.Source: D. Efficiency Analysis · p. 11

Accuracy and Inference Time

  • 8.1 seconds on WorldSense pairs with 40.8 accuracy; VideoScout also leads LVBench, MLVU, and MINERVA at 45.1, 51.0, and 35.4.The corresponding inference times are 35.1, 22.3, and 33.6 seconds on LVBench, MLVU, and MINERVA.Source: D. Efficiency Analysis · p. 11

Adaptive Inference Cost

  • VideoScout's inference time adapts to benchmark demands, from 8.1 seconds on WorldSense to 35.1 seconds on LVBench.The authors contrast this with LongVT-7B and VideoTemp-o3-7B, whose inference times remain nearly fixed across long-video benchmarks.Source: D. Efficiency Analysis · p. 11

Interpretation

  • VideoScout commits early when a short clip provides sufficient evidence and keeps exploring or rewatching when answers depend on long-range or fine-grained cues.The authors report that its higher cost on difficult long videos is consistently converted into higher accuracy.Source: D. Efficiency Analysis · p. 11

E. Qualitative Study

VideoScout's accuracy remains strong across video-duration buckets, and a qualitative example shows it revisiting uncertain evidence before answering. Its viewing actions and reasoning turns also vary with video length, unlike those of the untrained agentic variant.

E. Qualitative Study

  • After spotting a partly cut-off warning sign at 4×, VideoScout rewatches chunk 29 at 1×, reads the original text, and selects the correct answer.The example illustrates an observe–reflect–adjust loop in which the agent questions whether its current evidence is sufficient.Source: E. Qualitative Study · p. 12, Figure 7 · p. 12

F. Further Analysis

  • VideoScout achieves the highest accuracy in every duration bucket and does not degrade on the longest videos, where its score exceeds that on the shortest videos.The two agentic baselines perform worse on videos longer than thirty minutes than on videos under three minutes.Source: F. Further Analysis · pp. 12–13, Figure 8 · p. 13
  • As videos grow longer, VideoScout uses less 1× viewing, more 4× viewing, more reasoning turns, and a higher proportion of rewatches.Rewatching is rare on short videos and rises on medium and long videos, where fast-forwarding may have missed fine-grained evidence.Source: F. Further Analysis · p. 13, Figure 9 · p. 13
  • The untrained agentic variant stays almost entirely at 1×, barely rewatches, and uses fewer reasoning turns, indicating that its pace does not adapt to video length.The authors report that it tends to commit after only a few turns rather than continuing to gather evidence.Source: F. Further Analysis · p. 13

G. Limitations

VideoScout reports strong performance under a bounded per-turn visual context, but its textual memory, limited rewatch range, and visual-only input constrain use. The authors propose broader modality coverage, long-range revisiting, and additional tools.

G. Limitations

  • On very long trajectories, rewritten textual memory may let inaccurate observations from early turns influence later reasoning.The memory keeps context compact, but the authors note this risk.Source: G. Limitations · p. 13
  • Rewatch is limited to the most recent fast-forward window, preventing verification of early decisive evidence from a much later position.Fig. 10 illustrates the agent being unable to return to early chunks.Source: G. Limitations · p. 13, Figure 10 · p. 14
  • Because VideoScout uses visual input only, questions depending on speech or ambient sound are beyond its present scope.Source: G. Limitations · p. 13

V. CONCLUSION AND FUTURE WORK

  • VideoScout attains strong performance on long video understanding, reasoning, and cross-domain benchmarks while keeping inference efficient under a bounded per-turn visual context window.Source: §V · p. 14
  • Future directions include audio perception, long-range revisiting, and tools such as object grounding, OCR, and external knowledge retrieval.Source: §V · p. 14

Related papers

Loading 2609.15606v1…