PaperlayerView exact revision on arXiv

Source-linked AI summary

VAMR: Multi-Question Agentic Reasoning for Efficient Long-Form Video Understanding

Runquan Gui, Hanzhu Chen, Zehao Wang, Hanxin Zhu, Xin Li, Zhibo Chen

arXiv:2610.11171v1cs.CVcs.AI

TL;DR

Long-form video questions often concern different aspects of the same recording, but existing agents typically explore and build memory through separate question-specific trajectories. VAMR coordinates questions through a shared trajectory using question-conditioned perception, layered memory, and QHPO. Across three benchmarks, it achieves the highest accuracy and fewest reasoning rounds; on LVBench, it exceeds VideoARM by 4.3 points while substantially reducing processing.

  • Problem

    Existing video agents handle questions about the same recording in separate trajectories, repeating exploration and memory construction rather than jointly acquiring reusable evidence.

    Source: §1 · pp. 1–2

  • Method

    VAMR coordinates questions in one shared tool-use trajectory, combining question-conditioned perception and layered memory with QHPO for questions that progress and finish at different rounds.

    Source: §1 · p. 2

  • Results

    Across LVBench, Video-Holmes, and LongVideoBench, VAMR achieves the highest accuracy and fewest reasoning rounds; on LVBench, it exceeds VideoARM by 4.3 points.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Under the unified evaluation setting, shared video reasoning reduces repeated interaction without sacrificing answer quality across diverse long-video settings.

    Source: §6 · p. 10

    VAMR focuses on settings where multiple questions examine different aspects of the same video.

    Source: A LIMITATIONS · p. 14

Abstract

from arXiv · show

Long-form video understanding often involves multiple questions about different aspects of the same recording. Yet existing video agents typically process each question through an isolated tool-use trajectory. This repeatedly restarts video exploration and memory construction, missing opportunities to acquire evidence jointly and progressively build a shared understanding that supports the complete question set. We introduce \textbf{VAMR} (\textbf{V}ideo \textbf{A}gent for \textbf{M}ulti-Question \textbf{R}easoning), which coordinates all questions about a video through one shared tool-use trajectory. At each round, a persistent policy model can invoke tools for one or more unresolved questions and submit answers for questions with sufficient evidence. Question-conditioned visual perception retrieves fine-grained clues for several questions in one call, while layered multi-question memory integrates reusable context into a shared video story and preserves separate evidence for individual questions. After supervised fine-tuning initializes this interaction protocol, we propose question-horizon policy optimization (\qhpo) to optimize shared trajectories in which questions progress and finish at different rounds. Specifically, a question-level critic estimates the value of each active question, while round alignment maps each question advantage to the rounds that directly serve it before the aligned advantages are aggregated to optimize the shared actor. Across LVBench, Video-Holmes, and LongVideoBench, VAMR achieves the highest accuracy overall and the fewest reasoning rounds among iterative methods. On LVBench, it reaches 62.1\% accuracy, exceeding VideoARM by \textbf{4.3} points while reducing reasoning rounds and processed frames by \textbf{85.9\%} and \textbf{61.4\%}.

1 INTRODUCTION

Video agents typically handle each question about a recording in a separate reasoning loop, despite opportunities to reuse evidence. VAMR coordinates questions in one shared trajectory, and experiments report improved accuracy and efficiency across three benchmarks.

1 INTRODUCTION

  • Existing agents focus on one question at a time, repeating video exploration and memory construction instead of jointly guiding evidence acquisition.Questions about one recording may concern distinct events, dialogue, and temporal relationships, and one inspected segment may provide evidence for several questions.Source: §1 · pp. 1–2
  • 62.1% LVBench accuracy exceeds VideoARM by 4.3 points, while reasoning rounds and processed frames fall by 85.9% and 61.4%.VAMR also achieves the highest accuracy and fewest reasoning rounds on LVBench, Video-Holmes, and LongVideoBench.Source: §1 · p. 2
  • VAMR replaces independent per-question loops with one shared trajectory that coordinates reasoning across the complete question set.The paper presents this as the first video agent to coordinate multi-question reasoning within a shared trajectory, to the authors’ knowledge.Source: §1 · p. 2
  • QHPO estimates advantages per question, aligns them to rounds that directly serve each question, and aggregates them to train the shared actor.It targets shared trajectories where questions progress and terminate at different rounds.Source: §1 · pp. 2–3
  • Ablations and controlled analyses examine the shared architecture and QHPO, with reported gains across question densities and evidence-overlap quartiles.The reported benefit grows as more questions are coordinated in one loop.Source: §1 · p. 3

2 RELATED WORK

Prior long-video agents adaptively acquire evidence but organize retrieval and memory around individual queries. VAMR instead coordinates distinct questions in a shared trajectory, and QHPO aligns training credit with the questions and rounds served.

Long-form video reasoning

  • Prior agentic video methods organize policy state, adaptive retrieval, and stopping around individual queries; VAMR jointly schedules question-specific evidence needs in one persistent loop.Its observations propagate through both shared and question-specific memory.Source: §2 · p. 3

Cross-question information use

  • Unlike one-pass cross-question methods or UniVA’s composite objective, VAMR keeps questions distinct, jointly directs tool use, retains separate evidence, and allows asynchronous termination.The authors state that, to their knowledge, VAMR is the first video agent to coordinate reasoning across multiple questions within a shared trajectory.Source: §2 · p. 3

Policy optimization for tool-using agents

  • Existing tool-agent training associates a trajectory with one task outcome, whereas QHPO aligns question-level values to rounds that directly serve each question before updating the shared actor.The cited approaches address intermediate values or localized advantages, but not joint actions advancing questions with different termination times.Source: §2 · p. 3

3 VAMR: VIDEO AGENT FOR MULTI-QUESTION REASONING

VAMR treats questions about one video as concurrent objectives in a shared sequential process. Its persistent policy coordinates retrieval and answers, while question-conditioned perception and layered memory combine reusable context with question-specific evidence.

3.1 MULTI-QUESTION VIDEO REASONING

  • One shared policy and memory jointly select actions for active questions, with answered questions leaving the window and backlog questions taking their place until all are answered.The formulation makes coordination part of the trajectory, not merely a batching rule; the objective remains average accuracy across questions.Source: §3.1 · p. 4

3.2 VAMR FRAMEWORK

  • The QA-inspector conditions visual retrieval on selected question-option pairs, producing reusable segment context and distinct clues for multiple questions in one call.This supports cross-question reuse without collapsing individual evidence requirements.Source: §3.2 · p. 4
  • Layered memory combines a shared story with independent clue memories, and buffers recent observations before consolidation.When questions transition, new questions inherit the accumulated shared story but begin with their own clue memory.Source: §3.2 · pp. 4–5

4 LEARNING THE VAMR POLICY

VAMR policy learning first initializes a shared multi-question action protocol with supervised fine-tuning, then uses QHPO to improve retrieval and stopping across shared trajectories. QHPO estimates outcomes per question and aligns question advantages with the rounds that directly serve them before optimizing the shared actor.

4.1 TWO-STAGE SFT INITIALIZATION

  • SFT examples pair supported answer submissions and tool calls with generated reasoning, removing privileged answers and evidence before reasoning generation.Privileged information guides action selection but does not enter the policy input or reasoning trace.Source: §4.1 · p. 5

4.2 QUESTION-HORIZON POLICY OPTIMIZATION

  • QHPO optimizes retrieval and stopping decisions using complete shared-loop trajectories sampled from the current policy.It follows SFT initialization of the joint action protocol.Source: §4.2 · p. 5, §4 · p. 5
  • A question-level critic estimates each active question’s expected success because questions in the same shared state can have different probabilities of success.The critic is trained from each question’s terminal reward.Source: §4.2 · p. 6
  • QHPO retains each question’s advantage only for rounds whose actions directly serve it, then averages aligned advantages into a round-level signal for the shared actor.Actor optimization follows clipped PPO; the aggregated signal remains for the complete joint action.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments evaluate VAMR’s accuracy and efficiency across three video benchmarks, then examine its architecture, training, and behavior as question density and evidence overlap vary.

5.1 EXPERIMENTAL SETUP

  • Training uses CG-Bench, while evaluation covers LVBench, Video-Holmes, and LongVideoBench under a unified comparison of accuracy, Rounds/Q, and Frames/Q.LVBench averages 15.04 questions per video and 68.4-minute videos, the highest density and longest videos in the evaluation suite.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • Across three benchmarks, VAMR achieves the highest accuracy and fewest reasoning rounds; on LVBench, it reaches 62.1 accuracy and beats VideoARM by +4.3 points.On LVBench, it also reduces Rounds/Q by 85.9% and processed frames by 61.4%.Source: Table 1 · p. 7, §5.2 · p. 7
  • On Video-Holmes, VAMR gains 4.4 points over the trained independent-loop counterpart while reducing reasoning rounds and processed frames by 54.8% and 29.0%; on LongVideoBench, the respective changes are +1.4 points, 57.1%, and 9.0%.The paper associates LongVideoBench’s smaller accuracy and visual-efficiency gains with its lower question density, while noting a remaining sequential-round advantage.Source: §5.2 · p. 7

5.3 ABLATION STUDIES

  • The vanilla shared loop raises accuracy from 45.2 to 49.6 while reducing Rounds/Q from 4.8 to 1.6; full VAMR reaches 56.8 accuracy.The QA-inspector and layered memory further raise accuracy to 53.9 and 52.3, respectively; together in full VAMR, they improve targeted clue retrieval and evidence organization.Source: §5.3 · p. 8, Figure 3 · p. 8
  • Full QHPO reaches 62.1 accuracy, versus 58.7 for PPO and 60.0 for QHPO without round alignment.Round alignment maps each question advantage to rounds whose actions serve that question, adding 2.1 points over QHPO without alignment under the same architecture, data, and optimization settings.Source: §5.3 · p. 8, Figure 3 · p. 8

5.4 ANALYSIS

  • With question density increased from 1 to 16, accuracy rises from 58.1 to 62.8 while Frames/Q falls from 219.2 to 87.7.The analysis holds videos, questions, input information, and inference settings fixed while changing the number of questions handled in each shared loop.Source: §5.4 · p. 9, Figure 4 · p. 9
  • VAMR improves accuracy in every evidence-overlap quartile; even Q1, with 1.4% pair overlap, gains 7.0 points and reduces frames by 63.4%.Frame reduction does not increase monotonically with evidence overlap.Source: §5.4 · p. 9, Figure 4 · p. 9

6 CONCLUSION

VAMR coordinates questions about one video through a shared agent trajectory, using question-conditioned perception, layered memory, and QHPO. Across three benchmarks, it achieves the highest accuracy and fewest reasoning rounds in the unified evaluation.

6 CONCLUSION

  • VAMR coordinates questions about one video in a shared agent trajectory, using question-conditioned perception and layered memory to preserve reusable video context and question-specific evidence.Its policy can invoke tools and resolve questions at different times.Source: §6 · p. 9
  • QHPO uses question-level value estimation and direct round alignment to optimize shared trajectories where questions progress and finish at different times.Source: §6 · p. 9
  • Across three benchmarks, VAMR achieves the highest accuracy and fewest reasoning rounds; on LVBench, it beats VideoARM by 4.3 points while reducing rounds and processed frames by 85.9% and 61.4%.Ablations attribute the main efficiency gain to the shared loop, with question-conditioned perception, layered memory, and QHPO further improving accuracy.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies the documented protocols and implementation details. It also states VAMR’s scope, situates it among related methods, and describes agent execution and supervised-example construction.

REPRODUCIBILITY STATEMENT

  • Appendices C–I document the execution protocol, SFT construction, optimization hyperparameters, dataset partitions, baseline implementations, metric accounting, and ablation and analysis settings and results.Source: REPRODUCIBILITY STATEMENT · p. 11

A LIMITATIONS

  • VAMR’s stated primary scope is settings where multiple questions examine different aspects of the same video.The paper identifies this setting in video-QA benchmarks and applications such as content moderation and security inspection.Source: A LIMITATIONS · p. 14

B MORE BACKGROUND

  • Unlike methods focused on individual queries, VAMR coordinates joint actions across asynchronously resolved questions; QHPO aligns each question’s advantage with rounds that serve it.Source: B MORE BACKGROUND · p. 14

C AGENTIC FRAMEWORK DETAILS

  • The agent operates on a five-question active window, allows up to six tool calls per round, and consolidates reusable context into shared story memory while routing question-specific evidence to clue memories.The QA-inspector returns a shared story summary and per-question clues; fine-grained retrieval revisits localized intervals, while ASR returns transcripts.Source: C AGENTIC FRAMEWORK DETAILS · p. 15

D SFT CONSTRUCTION DETAILS

  • SFT examples pair public-state reasoning with executed joint actions: a privileged first stage selects actions using answers and evidence annotations, then a second stage removes those annotations and generates reasoning.Examples relying on hidden annotations or unavailable evidence are discarded.Source: D SFT CONSTRUCTION DETAILS · p. 15

E QHPO TRAINING DETAILS

QHPO trains the shared-loop policy using complete video trajectories, question-level rewards and values, and action-based attribution. Its optimization setup combines an initial critic pass with subsequent joint actor–critic updates.

E QHPO TRAINING DETAILS

  • QHPO uses three fresh on-policy passes over shared-loop trajectories: one critic warm-up with the actor frozen, then two joint actor–critic passes.Each update collects eight video trajectories and optimizes four-video minibatches.Source: E QHPO TRAINING DETAILS · p. 16
  • A question receives reward 1 for a correct submitted answer and 0 otherwise; gold answers are used only after rollout, not as actor or critic input.Source: E QHPO TRAINING DETAILS · p. 16
  • For each question, GAE uses zero rewards before resolution, its outcome reward at resolution, and zero value after resolution.The question-level advantage is computed across the rounds when that question is active.Source: E QHPO TRAINING DETAILS · p. 16
  • Validated answer submissions serve their submitted question, QA-inspector calls serve their target questions, and ASR calls serve all questions active that round.Advantages are retained only for rounds attributed to a question; rounds serving no questions receive no actor update.Source: E QHPO TRAINING DETAILS · p. 16

F DATASET STATISTICS

The dataset statistics section describes the CG-Bench training and validation partitions and characterizes the held-out evaluation benchmarks. It also documents the data-overlap audit and duration-measurement sources.

F DATASET STATISTICS

  • VAMR is trained and validated only on CG-Bench, whose 1,219 videos are partitioned by video identifier so questions from the same video do not cross splits.Table 2 reports the SFT, QHPO, and validation partitions.Source: F DATASET STATISTICS · p. 16, Table 2 · p. 16
  • LVBench averages 15.04 questions per video, more than Video-Holmes and LongVideoBench; it contains hour-long videos with dense question groups.LongVideoBench covers diverse long-context video tasks, while Video-Holmes focuses on multi-step reasoning.Source: F DATASET STATISTICS · p. 16, Table 3 · p. 17
  • Video-Holmes durations are estimated from the final timestamp in released segment annotations because its question file lacks video durations.CG-Bench and LongVideoBench durations use released metadata, while LVBench follows official dataset statistics.Source: F DATASET STATISTICS · p. 17
  • The released CG-Bench source video identifiers do not overlap with the 103 LVBench, 270 Video-Holmes test, or 753 LongVideoBench validation videos.Source: F DATASET STATISTICS · p. 17

G MAIN EXPERIMENT DETAILS

The main-experiment details specify baseline evaluation, cost accounting, and run-to-run variability. A complementary comparison reports VAMR’s accuracy under native model configurations, which differ across methods.

G MAIN EXPERIMENT DETAILS

  • The shared protocol gives baselines all questions and options as read-only context, while answers, tool calls, and memory updates remain restricted to the target question.Direct controls use 32 sampled frames; tool controls and matched-budget independent-loop policies use private per-question memory.Source: G MAIN EXPERIMENT DETAILS · pp. 17–18
  • Table 4 reports accuracy means and standard deviations across three inference runs, quantifying run-to-run variability on fixed evaluation sets.Source: Table 4 · p. 17, G MAIN EXPERIMENT DETAILS · p. 17
  • Rounds/Q amortizes valid joint reasoning rounds per question, while Frames/Q amortizes visual-model frames and recounts frames reprocessed in later calls.Model Calls/Q separately counts external model invocations per question; DVD and VideoHV-Agent make no external call on the final answer round.Source: G MAIN EXPERIMENT DETAILS · pp. 17–18, Table 5 · p. 18
  • Video-R1 and DeepVideo-R1 use 32 sampled frames without iterative actions; other baselines use temporal selection, video databases, captioning, or per-question tool loops.The comparison details FrameThinker, TimeSearch-R, DVD, VideoHV-Agent, and VideoARM separately.Source: G MAIN EXPERIMENT DETAILS · pp. 18–19
  • 80.4 LVBench, 62.5 Video-Holmes, and 72.9 LongVideoBench accuracy are VAMR’s native-stack results; VideoARM reports 79.7 on LVBench and 73.7 on LongVideoBench.Table 6 compares native model configurations; prior-method values are published results.Source: Table 6 · p. 19, G MAIN EXPERIMENT DETAILS · p. 19
  • Native configurations differ in backbones, frames, subtitles, tools, and inference budgets, so the Table 6 comparison uses non-matched method settings.Source: G MAIN EXPERIMENT DETAILS · p. 19

H ABLATION EXPERIMENT DETAILS

The ablation details compare architecture components and policy-optimization choices under controlled settings. Architecture and training ablation results are reported in separate LVBench tables and visualized in Figure 3.

H ABLATION EXPERIMENT DETAILS

  • Tables 7 and 8 provide numerical results for the architecture and training ablations, respectively, and Figure 3 visualizes those results.Source: H ABLATION EXPERIMENT DETAILS · p. 19, Table 7 · p. 20, Table 8 · p. 20
  • The architecture ablation compares independent loops with shared-loop variants that add a QA-inspector, layered memory, or both, using the same model and evaluation settings.The vanilla shared variant uses a generic inspector and single-layer memory; independent loops reset context and memory for each question.Source: H ABLATION EXPERIMENT DETAILS · p. 19, Table 7 · p. 20
  • The training ablation holds the VAMR architecture and evaluation protocol fixed while comparing PPO with QHPO variants that use question-level advantages.With round alignment, an advantage is retained only for rounds whose validated joint action serves that question; without it, the advantage spans all rounds until resolution.Source: H ABLATION EXPERIMENT DETAILS · pp. 19–20, Table 8 · p. 20

I ANALYSIS EXPERIMENT DETAILS

The analysis details controlled question-density scaling and a separate gold-evidence-overlap analysis, including how questions, evidence intervals, and visual-tool workload are handled.

I ANALYSIS EXPERIMENT DETAILS

  • 768 LVBench questions from 48 videos are evaluated at group sizes 1, 2, 4, 8, and 16, with the same 16 question-option pairs exposed at every density.Questions in each group share one loop and persistent memory; at most five are active, with the rest entering from a backlog.Source: I ANALYSIS EXPERIMENT DETAILS · p. 20, Table 9 · p. 21
  • The evidence-overlap analysis compares independent SFT+PPO loops with SFT+QHPO VAMR on 103 LVBench videos and 1,549 questions, defining each question’s gold evidence as its annotated intervals’ union.Source: I ANALYSIS EXPERIMENT DETAILS · p. 20
  • 1,547 questions have valid evidence intervals; the other two are excluded only from overlap calculations, and reversed endpoints and timestamp formats are normalized.Source: I ANALYSIS EXPERIMENT DETAILS · p. 20
  • Frames/Q counts the full visual-tool workload, including repeated processing across separate calls; videos are quartiled by gold evidence-pair overlap, from lowest (Q1) to highest (Q4).Source: I ANALYSIS EXPERIMENT DETAILS · p. 21, Table 10 · p. 21
Loading 2610.11171v1…