PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning

Ye Liu, Kevin Qinghong Lin, Chang Wen Chen, Mike Zheng Shou

arXiv:2503.13444v3cs.CVcs.AI

TL;DR

Video reasoning must connect answers to interpretable evidence across time, but long videos challenge methods that cannot localize or revisit moments. VideoMind coordinates four specialized roles through a unified model with role-specific LoRA adapters. Across 15 benchmarks, it reports strong performance across three task families, with its 2B model surpassing GPT-4o and Gemini-1.5-Pro on several long-video benchmarks.

  • Problem

    Video reasoning requires understanding how visual appearances evolve over time, while static-image reasoning methods struggle to localize or revisit moments in long videos.

    Source: §1 · p. 1

  • Method

    VideoMind uses a planner, grounder, verifier, and answerer, implemented with role-specific LoRA adapters on a unified model for flexible role switching.

    Source: §1 · p. 1, §3.5 · p. 6

  • Results

    Across 15 benchmarks spanning Grounded VideoQA, Video Temporal Grounding, and General VideoQA, VideoMind performs strongly, and its 2B model surpasses GPT-4o and Gemini-1.5-Pro on several long-video benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VideoMind demonstrates the effectiveness of an agentic workflow with Chain-of-LoRA for evidence-based answers in long-form video reasoning.

    Source: §5 · p. 9

    The method requires careful optimization of individual designs and preparation of training data.

    Source: §5 · p. 9

Abstract

from arXiv · show

Videos, with their unique temporal dimension, demand precise grounded understanding, where answers are directly linked to visual, interpretable evidence. Despite significant breakthroughs in text-based reasoning with large language models, multi-modal reasoning - especially for videos - remains limited. In this work, we fill this gap by introducing VideoMind, a novel video-language agent for temporal-grounded video reasoning. Our method involves two key innovations: (1) We identify four essential capabilities for grounded video reasoning and propose a role-based agentic workflow, comprising a planner to coordinate roles, a grounder for temporal event localization, a verifier to assess event candidates, and an answerer for question answering. (2) To efficiently integrate these roles during inference, we propose a novel Chain-of-LoRA mechanism, where a unified base model with multiple LoRA adapters is leveraged to enable seamless role switching, balancing efficiency and flexibility. Extensive experiments on 15 benchmarks across Grounded VideoQA, Video Temporal Grounding, and General VideoQA tasks demonstrate the effectiveness of the proposed scheme in advancing video agent, test-time scaling, and long-form video reasoning. Code, models, datasets, and demos are available at https://videomind.github.io/.

1 INTRODUCTION

Video reasoning must track how visual events evolve over time, yet static-image reasoning methods struggle to localize and revisit moments in long videos. VideoMind addresses this with coordinated roles and a unified model using role-specific LoRA adapters, and reports strong results across three video-task families.

1 INTRODUCTION

  • Video’s temporal dimension requires tracking how appearances evolve, while static-image CoT methods struggle to localize or revisit earlier moments in long videos.The paper motivates an agent that decomposes problems, identifies and revisits relevant moments, then synthesizes observations.Source: §1 · p. 1
  • VideoMind assigns task decomposition and coordination to a planner, moment localization to a grounder, candidate assessment to a verifier, and response generation to an answerer.The grounder uses a timestamp decoder for temporal grounding.Source: §1 · pp. 1–2
  • Chain-of-LoRA implements the roles as adapters on one unified LMM backbone, allowing role switches without maintaining multiple full models.Role-specific capabilities can be trained separately on tailored datasets; inference switches among cached LoRA parameters.Source: §1 · p. 2
  • Across 15 benchmarks, VideoMind performs strongly on three task families, and its 2B model surpasses GPT-4o and Gemini-1.5-Pro on several long-video benchmarks.The paper also reports state-of-the-art performance on QVHighlights and Charades-STA.Source: §1 · p. 2

2 RELATED WORK

Prior video understanding work often lacks grounded correspondence and interpretability, while video reasoning remains challenging for multimodal and text-based approaches. VideoMind situates its approach among temporal grounding and inference-time search methods.

2 RELATED WORK

  • Video retrieval and captioning models often lack visually grounded correspondence and interpretability for long-form videos; temporal grounding targets precise localization for diverse queries.Regression-based localization models can fall short in textual interpretability, while prior multi-task or modular approaches may be sub-optimal or overly complex.Source: §2 · pp. 2–3
  • Multimodal models support generalized free-form question answering, but extending reasoning to videos remains an open challenge given their long-context nature.The passage contrasts visual-tool agents and text-based chain-of-thought approaches, including extensions to vision.Source: §2 · p. 3
  • Inference-time search methods refine outputs without changing model weights, but their potential in video understanding remains largely untapped.VideoMind uses moment-level search: a grounder generates candidates and a verifier selects the best correspondence.Source: §2 · p. 3

3 METHOD

VideoMind reasons over videos by adaptively calling a planner, grounder, verifier, and answerer, allowing it to revisit video segments at varying temporal and spatial resolutions. Its method specifies task-dependent role plans, a timestamp decoder, candidate verification, and role-specific LoRA adapters on a shared backbone.

3.1 PLANNER

  • The planner selects among three task-specific role plans and can rephrase underspecified queries to support moment localization.Grounded QA combines grounding, verification, and answering; retrieval uses grounding and verification, while straightforward or very short-video questions can be answered without grounding.Source: §3.1 · pp. 3–4

3.2 GROUNDER

  • The grounder uses a <REG>-conditioned timestamp decoder to predict normalized start and end times from video-frame and query representations.It pools visual tokens to one token per frame, fuses frame and query representations with a transformer, and uses a multilevel temporal feature pyramid for moments of varying lengths.Source: §3.2 · p. 5

3.3 VERIFIER

  • The verifier zooms in on the grounder’s top five candidates and selects the moment with the highest confidence for the answerer.It expands each candidate’s boundaries by 50%, judges whether the event occurs within them using binary responses, and scores candidates from the likelihoods of “Yes” and “No.”Source: §3.3 · p. 6, Figure 5 · p. 6

3.4 ANSWERER

  • The answerer responds using either the cropped moment when grounding is used or the whole video otherwise, without fine-tuning or architectural changes.Source: §3.4 · p. 6

3.5 CHAIN-OF-LORA

  • Chain-of-LoRA runs the roles on a shared LMM backbone and dynamically activates role-specific adapters, with an additional timestamp decoder used only by the grounder.This design supports flexible role switching while minimizing memory consumption and architectural modifications to the base model.Source: §3.5 · pp. 6–7

4 EXPERIMENTS

Experiments test VideoMind across diverse video-understanding tasks, compare Chain-of-LoRA with alternative role-integration strategies, and examine individual roles. Results cover grounded question answering, temporal grounding, and general VideoQA, with a visualization of the role sequence.

4 EXPERIMENTS

  • The experiments span 15 public benchmarks and ask whether VideoMind generalizes across tasks, how Chain-of-LoRA compares with other integration strategies, and what each role contributes.Appendix material is identified for benchmark settings, implementation details, and additional results.Source: §4 · p. 7

4.1 Q1: COMPARISON WITH STATE-OF-THE-ARTS

  • On CG-Bench, the 2B model beats all baselines on mIoU and rec.@IoU, while 7B sets a new acc.@IoU state of the art; NExT-GQA results cover grounding and QA.Source: §4.1 · pp. 7–8, Table 4 · p. 8
  • VideoMind surpasses LLM-based methods and is competitive with fine-tuned experts on temporal grounding; its General VideoQA design helps localize cue segments before answering.Source: §4.1 · p. 8, Table 5 · p. 8, Table 6 · p. 8

4.2 Q2: THE ADVANTAGES OF CHAIN-OF-LORA

  • Chain-of-LoRA offers the best effectiveness-efficiency balance: text-based CoT does not improve the base model, joint training is suboptimal, and distributing roles requires 4× the weights.Source: §4.2 · pp. 8–9, Table 7 · p. 9

4.3 Q3: KEY ABLATION STUDIES

  • The grounder slightly improves QA accuracy, the verifier adds 3.2 mIoU on Charades-STA, and planner-led grounding on 40% of samples raises accuracy from 69.2 to 70.0.Table 8 labels A, G, V, P, and G% as answerer, grounder, verifier, planner, and the percentage of samples processed with the grounder, respectively.Source: §4.3 · p. 9, Table 8 · p. 9

4.4 VISUALIZATION

  • The planner selects roles, the grounder proposes moments, the verifier selects a relevant segment, and the answerer reasons over the zoomed-in clip to produce an answer.Source: §4.4 · p. 9, Figure 6 · p. 9

5 CONCLUSION

The conclusion presents VideoMind as a four-role agent using Chain-of-LoRA for temporal-grounded video reasoning, with experiments supporting its effectiveness across three video task families. It also notes that individual designs require careful optimization and training-data preparation.

5 CONCLUSION

  • VideoMind combines a four-role agentic workflow with Chain-of-LoRA to switch flexibly among roles for temporal-grounded video reasoning.Source: §5 · p. 9
  • Experiments on Grounded VideoQA, Video Temporal Grounding, and General VideoQA demonstrate effectiveness, particularly for long-form video reasoning with evidence-based answers.Source: §5 · p. 9
  • The method requires careful optimization of individual designs and preparation of training data.Source: §5 · p. 9

ETHICS STATEMENT

The study focuses on algorithmic innovations for visual reasoning in large multimodal models and uses publicly available datasets. It notes potential LLM/LMM misuse risks but does not directly address deployment or generation.

ETHICS STATEMENT

  • The study focuses on algorithmic innovations for improving visual reasoning in large multimodal models.Source: ETHICS STATEMENT · p. 10
  • It involves no human subjects or private data, and its datasets are publicly available and widely adopted in the community.Source: ETHICS STATEMENT · p. 10
  • The authors acknowledge bias propagation and harmful content generation as potential LLM/LMM misuse risks, while noting that the study does not directly address deployment or generation.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points to implementation and evaluation materials and says code, checkpoints, data, and training logs are open-sourced. The appendix lists further inference, benchmark, and analysis material.

REPRODUCIBILITY STATEMENT

APPENDIX

  • The appendix provides inference-pipeline and implementation details, plus additional experiments, analysis, and discussions.Source: APPENDIX · p. 17

1. Prompt Templates

  • The appendix contents list prompt templates as a separate item.Source: §1 · p. 17

A.1 INFERENCE PIPELINE

  • Given a video and question, the planner selects roles; the grounder proposes moments, the verifier scores zoomed-in candidates, and the answerer returns an answer with a temporal moment.Source: §A.1 · p. 17

A.2 IMPLEMENTATION DETAILS

The implementation uses Qwen2-VL base models with separately trained role-specific LoRAs, loaded together to support role switching at inference. The benchmark set spans three task types and video durations from seconds to over an hour.

A.2 IMPLEMENTATION DETAILS

  • Planner, grounder, and verifier use rank-64, alpha-64 LoRAs on 2B and 7B Qwen2-VL; roles are trained separately and loaded together for inference-time switching.The timestamp decoder hidden size is 256, and grounding uses NMS at IoU = 0.75 to reduce duplicate moments.Source: §A.2 · p. 17
  • The evaluation benchmarks cover Grounded VideoQA, Video Temporal Grounding, and General VideoQA, with video durations from seconds to more than one hour.Table 10 reports MultiHop-EgoQA performance; Sent. Sim. denotes sentence similarity computed by all-MiniLM-L6-v2.Source: Table 9 · p. 18, Table 10 · p. 18

B.1 BENCHMARKS AND SETTINGS

The evaluation spans benchmarks for grounded question answering, temporal grounding, and general video question answering. The described datasets include long-form videos, temporal annotations, and causal or temporal reasoning tasks.

B.1 BENCHMARKS AND SETTINGS

  • The benchmark suite covers 15 datasets across Grounded VideoQA, Video Temporal Grounding, and General VideoQA, with durations from seconds to over an hour.The cited table describes the breadth of benchmark tasks and durations.Source: §B.1 · p. 18
  • CG-Bench contains 1.2K manually curated videos lasting 10–80 minutes and 12K QA pairs, with perception, reasoning, and hallucination questions.Its white-box and black-box clue-based assessments evaluate whether answers rely on accurate video reasoning.Source: §B.1 · p. 18
  • ReXTime evaluates complex temporal reasoning, while NExT-GQA contains causal and temporal questions paired with annotated video segments.ReXTime has 921 validation and 2.1K test samples; NExT-GQA has 10.5K manually labeled QA pairs and supports both multiple-choice and open-ended tasks.Source: §B.1 · p. 18
  • The TACoS and Ego4D-NLQ tables report temporal-grounding results, and the QVHighlights table reports fine-tuned grounding results.The QVHighlights results are identified as fine-tuned in the table caption.Source: Table 11 · p. 19, Table 12 · p. 19
  • The grounding benchmarks include indoor activity clips with temporal annotations and dense captions used as queries for localizing moments.Charades-STA has 10K videos averaging 30.1 seconds and 16K annotations; ActivityNet-Captions has 20K untrimmed videos and 849 hours of footage.Source: §B.1 · p. 19

B.2 MORE EXPERIMENTAL RESULTS

VideoMind is evaluated on multi-hop grounded QA, temporal localization and grounding, and general video QA. The reported comparisons show strong results across these settings, including a new state-of-the-art claim for VideoMind-2B against prior temporal-grounding methods.

Multi-Hop Grounded Question Answering

  • On MultiHop-EgoQA, VideoMind surpasses all open-source baselines and remains competitive with GPT-4o across grounding and QA metrics.The authors attribute this result to producing multiple candidate moments in one grounding step, which can capture the benchmark’s multi-hop evidence.Source: §B.2 · p. 19

Reasoning Temporal Localization

  • VideoMind-7B outperforms the fine-tuned LITA-13B baseline by a considerable margin on zero-shot reasoning temporal localization.This task requires inferring the event using world knowledge because queries do not directly describe the target moment.Source: Table 14 · p. 20, §B.2 · p. 20

Video Temporal Grounding

  • On QVHighlights, VideoMind-2B significantly outperforms all previous methods and sets a new state of the art; on TACoS it outperforms UniVTG, but performs slightly worse on Ego4D-NLQ.Source: §B.2 · pp. 19–20

General Video Question Answering

Error Distribution

  • Figure 7 presents the error distribution for the 2B and 7B variants on ReXTime (Chen et al., 2024c) and NExT-GQA (Xiao et al., 2024), separating planning, grounding, verification, and answering errors.Source: Figure 7 · p. 21

Comparison with Text-based Reasoning Models

  • VideoMind significantly outperforms both representative pure text-based video reasoning baselines on all reported benchmarks.The authors characterize the comparison as evidence favoring vision-centric reasoning for long or complex video reasoning tasks.Source: §B.2 · p. 21, Table 17 · p. 21

B.3 MORE DETAILED ANALYSIS

The analyses test timestamp modeling, temporal features, verification, planning, efficiency, and robustness, alongside integrations and prompt templates. They report gains from the timestamp decoder and verifier, identify grounding as a major error source, and describe inference-time and cross-model results.

Timestamp Modeling Designs

  • The timestamp decoder provides the strongest temporal grounding among the tested designs, decouples continuous timestamp modeling from discrete token prediction, and supports multiple moments with confidence scores.The authors also cite direct L1 regression supervision as improving time reasoning and stabilizing training.Source: §B.3 · p. 22

Effect of the Temporal Feature Pyramid

  • Adding temporal feature-pyramid levels consistently improves zero-shot grounding across all metrics on Charades-STA, supporting robustness to moments of different lengths.The baseline predicts directly from the last-layer transformer outputs.Source: §B.3 · pp. 22–23, Table 20 · p. 22

Verifier Analysis

  • The verifier improves grounding, including 18.5% higher R@0.7 on the 2B variant; boundary expansion broadens its perceptual range, while special tokens improve awareness of precise boundaries.The verifier-style comparison defines expansion as widening temporal boundaries by 50%.Source: Table 22 · p. 22, Table 21 · p. 22, §B.3 · p. 23

Planner Reliability and Inference-Time Efficiency

  • 93% planning accuracy follows from incorporating both video and question inputs; auto-planning reduces average inference time from 9.53s to 8.07s per video, 7.8% faster than LongVILA-R1.The efficiency comparison was conducted on CG-Bench using one NVIDIA RTX 6000 Ada GPU.Source: Table 23 · p. 22, Table 24 · p. 22, §B.3 · pp. 23–24

Overall Robustness and Error Accumulation

  • Grounding errors make up roughly half of failures, while planning errors account for under 5% and verification and answering errors about 20% each; grounding IoU correlates clearly with QA accuracy on ReXTime, less so on NExT-GQA.The error breakdown covers ReXTime and NExT-GQA; the correlation analysis distinguishes the more temporal-related ReXTime from the more reasoning-related NExT-GQA.Source: Figure 8 · p. 22, §B.3 · p. 24

Controlled Experiments on Closed-source APIs

  • In controlled comparisons on 300 sampled QA pairs from each of MLVU and LVBench, VideoMind-7B outperforms GPT-4o and Gemini-1.5-Pro under aligned hyperparameters.The experiments align frame rate, maximum frame count, resolution, and decoding hyperparameters.Source: Table 25 · p. 23, §B.3 · p. 24

Integration with Open- and Closed-source LMMs

  • Chain-of-LoRA results improve with stronger open-source base models, while prompting alone also boosts GPT-4o, GPT-5, and Gemini-2.5-Pro on the sampled long-video benchmarks.The authors present the prompt-based results as evidence that the multi-role pipeline can enhance long-video reasoning without role-specific model designs or training.Source: Table 26 · p. 23, Table 27 · p. 23, §B.3 · p. 24

C.1 PROMPT TEMPLATES; D THE USE OF LLMS STATEMENT

  • The prompt templates specify tool roles and planning sequences, while the authors report using LLMs only to polish manuscript language and check grammar.The planner selects tools, the grounder localizes a queried moment, and the verifier checks segment boundaries.Source: §C.1 · p. 25, D THE USE OF LLMS STATEMENT · p. 26
Loading 2503.13444v3…