Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning

• VITAL achieves superior performance across 11 benchmarks in long video understanding, video question answering, and temporal grounding, with state-of-the-art performance on VUE-TR.Source: Figure 1 · p. 1, §1 · p. 2
• Text-only CoT reasoning can limit cross-modal interaction and increase hallucination, particularly for long videos or reasoning chains.Source: §1 · p. 1
• VITAL is an end-to-end agentic framework whose visual toolbox samples frames on demand, helping the model focus on critical temporal segments during multimodal reasoning.The framework uses a cold-start phase followed by reinforcement learning, jointly optimizing temporal grounding, video question answering, and grounded question answering.Source: §1 · pp. 1–2
• The authors construct MTVR-CoT-72k for supervised fine-tuning and MTVR-RL-110k for reinforcement learning to support multi-task video reasoning.Source: §1 · p. 2
• DGRPO adjusts reward scales according to task and sample difficulty to mitigate difficulty imbalance in multi-task reinforcement learning.The paper says this adaptive balancing supports more stable training and improved generalization.Source: §1 · p. 2

2.1 Reasoning-Enhanced MLLMs

• Prior GRPO-style RL post-training enhances MLLM reasoning across image and video tasks, while VITAL uses multimodal rather than text-based CoT for video understanding [20].Related work spans mathematical and scientific image VQA [26] [53], image segmentation and grounding [2] [46] [47] [54] [66] [67], video reasoning VQA [8] [13] [38] [68], and video spatial or temporal grounding [18] [39] [52] [72].Source: §2.1 · pp. 2–3

2.2 Tool-Augmented LLMs

• Following tool-augmented LLM work, VITAL targets long-video reasoning with video grounding and clipping tools rather than image-focused tools [20].Image-task approaches include visual evidence for multimodal CoT in FAST [60] and MVoT [35], and image zoom-in, sketching, detection, and segmentation in DeepEyes [94] and OpenThinkImg [59].Source: §2.2 · p. 3

2.3 Long Video Understanding

• Long-video approaches use feature extraction, temporal selection, token compression, context extension, or dynamic frame sampling; VITAL instead proposes multimodal tool-augmented RL for efficient, accurate reasoning.Examples include iterative temporal selection in MIST [16] and SEVILA [87], long-context extension in LongVA [90] and LongVILA [7], and course-to-fine dynamic sampling [22] [71]; the passage notes that prior approaches can be computationally intensive.Source: §2.3 · p. 3

3.1 Tool-Augmented Learning Framework

• VITAL iteratively decides whether to call a visual tool or answer directly, forming multimodal CoT; its video-clipping tool returns densely sampled frames from a requested time range.Tool execution supplies a new video result for the next reasoning round, and the process ends when the model outputs its answer.Source: §3.1 · p. 4

3.2 Multi-Task Video Reasoning Training Data

• MTVR-CoT-72k and MTVR-RL-110k provide SFT and RL data for temporal grounding, VQA, and grounded VQA.The four subsets are MTVR-CoT (54k), MTVR-RL (94k), MTVR-CoT-Tool (18k), and MTVR-RL-Tool (16k); rollout filtering excludes samples where all or none of the rollouts pass.Source: §3.2 · p. 4

3.3 Difficulty-aware GRPO Training

• DGRPO combines accuracy, strict-format, and successful-tool-call rewards, then adjusts temporal-grounding IoU by task difficulty and sample weights by mean rollout reward.The method targets task-wise imbalance, where easier tasks improve faster, and sample-wise imbalance, where easier samples become more prevalent; sample weights range from 0.5 to 1.Source: §3.3 · p. 5
• DGRPO is reported more stable than GRPO and better on LongVideo-Reason and VidChapters-7M, while retaining good accuracy on Video-MMMU and Charades-STA.The comparison is reported for challenging long-video benchmarks and also notes retained accuracy on the other two benchmarks.Source: Table 1 · p. 5, §3.3 · p. 5

4.1 Experimental Setup

• VITAL-7B uses a Qwen2.5-VL-7B-pretrained language model and is evaluated on long-video QA, reasoning, and grounding benchmarks, plus complex reasoning, short-video grounding, and grounded VQA tasks.Training uses four one-epoch stages totaling 640 GPU hours; the model incorporates a visual encoder and merger projector.Source: §4.1 · pp. 5–6

4.2 Main Results

• LongVideo-Reason accuracy reaches 79.3% versus 67.9% for the previous best open-source model, while VidChapters-7M R@0.5 reaches 34.7% versus 27.4%.The authors report state-of-the-art performance on long-video question answering and temporal grounding, with particularly pronounced gains in long-video scenarios.Source: Table 2 · p. 6, §4.2 · p. 6
• VITAL-7B consistently outperforms strong baselines on VSI-Bench, Video-MMMU, and MMVU, and excels on short-video temporal grounding and grounded VQA benchmarks.The MMVU evaluation uses its multiple-choice split; the authors report state-of-the-art performance on NExT-GQA and ReXTime.Source: §4.2 · p. 6, Table 3 · p. 6, Table 4 · p. 7

4.3 Ablation Study

• DGRPO raises the average score from 50.3 to 52.1 on difficult long-video tasks while keeping short-video perception ability unchanged.The authors attribute the improvement to difficulty-aware reward balancing across diverse task complexities.Source: §4.3 · p. 7
• Tool-augmented RL improves long-video perception and yields substantial gains across benchmarks, while toolbox deltas indicate consistent benefits across video reasoning tasks.The comparison is between rows 6 and 7 of Table 1; the reported toolbox deltas are in Tables 2–4.Source: Table 2 · p. 6, Table 3 · p. 6, §4.3 · p. 7
• Combining temporal grounding, reasoning VQA, and grounded VQA data produces the best overall performance, raising the average score from 37.9% without training data to 57.1%.The authors report that this three-task combination provides synergy for video reasoning and temporal grounding.Source: §4.3 · p. 7, Table 5 · p. 7
• In zero-shot temporal grounding tests, clip caption and clip QA tools do not improve GPT or Gemini results and can significantly reduce mIoU; video clipping reduces hallucination.Because the models were not optimized for visual tools in this setting, the authors adopt video clipping by default and use four-stage training to enhance tool calling.Source: §4.3 · p. 7, Table 6 · p. 7

4.4 Qualitative Analysis

• Text-based CoT accumulates errors and hallucinations, whereas multimodal CoT uses visual evidence to determine target-event temporal boundaries more precisely.The qualitative comparison reports a lower IoU score for text-based reasoning.Source: §4.4 · p. 7

5 Conclusion

• VITAL equips MLLMs with tool-augmented multimodal reasoning for long videos, mitigating hallucination and enhancing cross-modal interaction.The authors highlight tool-augmented multimodal reasoning as important for future long video understanding research.Source: §5 · p. 7
• The work constructs two high-quality multi-task video reasoning datasets and proposes DGRPO to address difficulty imbalance in multi-task reinforcement learning.Source: §5 · p. 7

Supplementary Material

• The supplement presents extended ablation studies and case analyses to evaluate VITAL’s effectiveness and robustness.Source: Supplementary Material · p. 12
• It includes experiment details and dataset-construction information.Source: Supplementary Material · p. 12
• It discusses limitations of the current work and outlines potential future directions.Source: Supplementary Material · p. 12
• It provides experimental code with detailed documentation for reference and reproducibility.Source: Supplementary Material · p. 12

A Method Implementation Details

• The supplement provides additional implementation details for VITAL, including its visual toolbox and DGRPO.Source: A Method Implementation Details · p. 12
• The method-details section first explains gtool implementation and then describes DGRPO.Source: A Method Implementation Details · p. 12

A.1 Tool Implementation Details

• The visual toolbox provides clip captioning, clip question answering, and video clipping to return visual tokens for a selected segment.Captioning and QA take clip start and end timestamps; QA also takes a natural-language question. The tools’ input and output formats are summarized in Table 7.Source: §A.1 · p. 12, Table 7 · p. 12
• Tool calls use JSON strings inside <tool call> tags, parsed under the Hermes-style protocol [1] [64], with JSON responses or error messages.Video-token outputs include the corresponding token sequence; failures from malformed parameters or other errors return a JSON dictionary string.Source: §A.1 · p. 12

A.2 DGRPO Algorithm Details

• DGRPO combines accuracy, format, and tool rewards; format reward is 0.5 for either exact response structure, while successful tool use earns 0.5.Malformed response structures receive zero format reward; the second-stage DGRPO without tools removes tool reward and lets format reward range from 0 to 1.Source: §A.2 · pp. 12–13
• Accuracy rewards use IoU for temporal grounding, EM for exact answers, Rouge, WER, and L1 for numerical answers.Rouge averages Rouge-1, Rouge-2, and Rouge-L; L1 is clamp(1−|ŷ−y|/|y|, 0, 1).Source: §A.2 · pp. 12–13
• DGRPO balances task difficulty with clamped linear transforms and sample difficulty with a piecewise-linear weight based on average rollout reward.It uses two settings for short- and long-video temporal grounding, leaves multiple-choice VQA task rewards unchanged, and transforms IoU for grounded VQA; sample-wise balancing softly penalizes easier samples as format and tool rewards converge.Source: §A.2 · p. 13

B Experimental Details

• Training and evaluation sample videos at 2 FPS, bound frames and pixels by video length, and print absolute frame timestamps using number prompts [78].Training image samples use max pixels = 448×448.Source: B Experimental Details · p. 14

B.1 Training Details

• The four training stages comprise cold-start SFT and RL without tools, followed by cold-start SFT and RL with tools, using long-video datasets in both tool stages.The authors report that starting tool training directly from the pretrained MLLM is inefficient and optimizes slowly because tool-augmented RL requires multi-round generation.Source: §B.1 · p. 14
• Training samples short videos at max frames = 64 and max pixels = 224×224, and long videos at max frames = 256 and max pixels = 112×112.Densely sampled tool-result clips use max frames = 64 and max pixels = 224×224; DGRPO rollouts use temperature = 1.0, top p = 1.0, and group size = 8.Source: §B.1 · p. 14

B.2 Evaluation Details

• VITAL-7B is evaluated on eleven challenging benchmarks.The evaluation configurations are presented in Table 11.Source: §B.2 · p. 14, Table 11 · p. 14
• All evaluation experiments use temperature 0.01, top p 0.001, maximum response length 1024, and at most two turns.These settings are used to ensure stable and reproducible results.Source: §B.2 · p. 14

C.1 Dataset Statistics

• The four MTVR subsets are MTVR-CoT (54k), MTVR-RL (94), MTVR-CoT-Tool (18k), and MTVR-RL-Tool (16k).Source: §C.1 · pp. 14–15
• Figure 8 presents video-duration and token-length distributions across subsets, with video durations varying significantly.The text also reports question and response token-length distributions.Source: §C.1 · pp. 14–15, Figure 8 · p. 15
• Figure 9 visualizes the most frequent words in each MTVR subset as word clouds.These highlight prevalent concepts and linguistic patterns.Source: Figure 9 · p. 15, §C.1 · p. 15

C.2 Data Filtering Criterion

• Samples with reward range ∆R_i,j ≤ 0.05 are discarded after the initial Qwen2.5-VL-7B [1] generates eight rollouts per sample.The remaining samples are selected for MTVR-RL and the next MTVR-CoT annotation step.Source: §C.2 · p. 15

C.3 Data Generation Prompts

• Multimodal CoT generation uses three rounds: reasoning, tool call, then reflection and a concluded answer; text-based CoT is also annotated.The reasoning MLLM is exemplified by Gemini 2.5 Pro [9].Source: §C.3 · p. 15
• For temporal grounding, the suggested tool range uses e′ = clamp(e + λ · |L − e| · rand(), 0, L), with λ set to 0.2 by default.The clamp keeps the result within the video duration range [0, L].Source: §C.3 · p. 15
• For other tasks such as video question answering, prompts omit the tool parameter suggestion, while reasoning mode is enabled in rounds one and three.Source: §C.3 · p. 16

C.4 Data Post-Process

• Post-processing yields MTVR-CoT-72k, with MTVR-CoT (54k) for stage 1 and MTVR-CoT-Tool (18k) for stage 3 training.Samples are excluded for incomplete reasoning, incorrect answers, or irrelevant content.Source: §C.4 · p. 16

D.1 Ablation Study on DGRPO Transformations

• Soft-penalty ω1(D) outperforms hard-penalty ω2(D) and continuous ω3(D) and ω4(D), with its wider difficulty-coefficient range offered as the explanation.The text says hard penalties remove easy samples’ contribution, harming training; ω1(D)’s advantage is attributed to format and tool rewards typically totaling 1 before varying from 1.0 to 2.0.Source: §D.1 · p. 16

D.2 Ablation Study on Cold Start

• Cold-start SFT alone yields few improvements without tools, but helps tool-augmented reinforcement learning; applying tool-augmented DGRPO directly after two-stage training impedes learning tool-call reasoning.The first two rows are tool-free baselines; additional tool training in rows 3 and 4 yields notable improvements, with row 4 demonstrating the benefit of cold-start SFT.Source: §D.2 · p. 16, Table 13 · p. 18

D.3 Fair Ablation of Tool Calling

• The initial tool-use comparison may be confounded by different final-stage video-token counts, motivating a fairer comparison matched to Exp. ⑦’s token budget.The additional two-stage experiment matches row 2 except for increased video resolution, setting max total pixels to 2×64×224×224.Source: §D.3 · p. 17
• In the last two rows of Table 13, VITAL-7B with tools (row 4) still significantly outperforms the improved baseline (row 5) by a large margin.The average number of successful tool calls in Exp. ⑦ is less than 1.Source: §D.3 · p. 17, Table 13 · p. 18

D.4 Ablation Study on Data Size

• Performance on both long- and short-video benchmarks consistently improves as the amount of stage-4 DGRPO training data increases, despite some metric fluctuations.Table 14 reports results at different steps during the last training stage; its data-size measure is the data already used during stage-4 DGRPO training.Source: §D.4 · p. 17, Table 14 · p. 18

E.1 Data Quality Analysis

• MTVR samples combine precise temporal annotations, detailed event descriptions, and diverse real-world scenarios, supporting fine-grained actions and complex reasoning across short and long videos.The passage presents these qualities as a foundation for robust video-language reasoning, including text-based CoT and tool-augmented multimodal CoT.Source: §E.1 · p. 18

E.2 Reasoning Quality Analysis

• Tool-augmented multimodal CoT enables VITAL-7B to reason more accurately and finely than text-only CoT in the case studies.The comparison is between VITAL-7B and VITAL-7B (w/o).Source: §E.2 · p. 18
• Combining visual and textual cues with tools for temporal grounding and evidence extraction supports precise event localization and deeper narrative understanding.The case studies describe these capabilities as benefits of systematically integrating multimodal cues.Source: §E.2 · p. 18
• The reported advantages concern challenging long-video question answering and temporal reasoning tasks.Source: §E.2 · p. 18

Limitations

• VITAL currently provides tools only for temporal grounding and question answering, limiting its ability to handle tasks such as spatial grounding.The framework mainly focuses on visual features and ignores audio, which may restrict overall video understanding.Source: F Limitations and Future Work · p. 18

Future Work

• Future work proposes expanding the toolbox and integrating visual and audio cues to support spatio-temporal grounding, semantic segmentation, and videos in complex scenarios.Source: F Limitations and Future Work · p. 18
• The authors also aim to develop more adaptive tool selection strategies to improve robustness and versatility in open-ended video understanding.Source: F Limitations and Future Work · p. 18