PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning

Jitesh Jain, Jialuo Li, Zixian Ma, Jieyu Zhang, Chris Dongjoo Kim, Sangho Lee, Rohun Tripathi, Tanmay Gupta, Christopher Clark, Humphrey Shi

arXiv:2512.13874v2cs.CV

TL;DR

Video reasoning models commonly answer in one turn, although video durations vary and long videos invite flexible, multi-turn reasoning. SAGE combines adaptive tool use with synthetic data and RL training; the authors report improvements for open-ended reasoning, including on long videos.

  • Problem

    Most long-video reasoning systems still use single-turn DIRECT prediction, while training agents to handle variable durations and open-ended questions with RL remains challenging.

    Source: §1 · p. 2

  • Method

    SAGE pairs an orchestrator that selects direct answers or multi-turn tool use with Gemini-2.5-Flash synthetic data and an RL recipe using LLM-judged rewards.

    Source: §1 · p. 2

  • Results

    SAGE’s RL recipe improves the SFT model by 4.1%, surpasses the base by 5.7%, and yields gains up to 14.6% on videos longer than 10 minutes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present SAGE as a proof of concept for practical AGENT systems for long-video reasoning beyond purely DIRECT approaches.

    Source: §5 · p. 9

Abstract

from arXiv · show

As humans, we are natural any-horizon reasoners, i.e., we can decide whether to iteratively skim long videos or watch short ones in full when necessary for a given task. With this in mind, one would expect video reasoning models to reason flexibly across different durations. However, SOTA models are still trained to predict answers in a single turn while processing a large number of frames, akin to watching an entire long video, requiring significant resources. This raises the question: Is it possible to develop performant any-horizon video reasoning systems? Inspired by human behavior, we first propose SAGE, an agent system that performs multi-turn reasoning on long videos while handling simpler problems in a single turn. Secondly, we introduce an easy synthetic data generation pipeline using Gemini-2.5-Flash to train the orchestrator, SAGE-MM, which lies at the core of SAGE. We further propose an effective RL post-training recipe essential for instilling any-horizon reasoning ability in SAGE-MM. Thirdly, we curate SAGE-Bench with an average duration of greater than 700 seconds for evaluating video reasoning ability in real-world entertainment use cases. Lastly, we empirically validate the effectiveness of our system, data, and RL recipe, observing notable improvements of up to 6.1% on open-ended video reasoning tasks, as well as an impressive 8.2% improvement on videos longer than 10 minutes.

1. Introduction

SAGE targets flexible video reasoning by combining multi-turn tool use for long-horizon tasks with direct answers for short-horizon problems. The paper introduces synthetic training data and an RL recipe, reporting gains across video durations.

1. Introduction

  • The paper asks how to train video reasoners under the AGENT paradigm, while most long-video systems still follow single-turn DIRECT reasoning, even with RL.The authors frame variable video durations and open-ended questions without verifiable rewards as challenges for RL post-training.Source: §1 · pp. 1–2
  • Gemini-2.5-Flash generates whole-video QnA pairs in one pass; manual checks of over 1,700 samples found a 5% error rate, with nearly 100× cost and 10× time savings.The comparisons are against human annotation and subclip-processing pipelines, respectively.Source: §1 · p. 2
  • SAGE uses SAGE-MM to choose between multi-turn tool use for long-horizon tasks and direct answers for short-horizon problems.Its tools include web search and speech transcription alongside temporal grounding, supporting visual, verbal, and external knowledge use.Source: Figure 1 · p. 1, §1 · p. 2
  • Training any-horizon agents requires direct answers for simple problems and multi-turn reasoning for harder ones, but open-ended tasks lack verifiable rewards.The introduction says some prior DIRECT approaches use only multiple-choice problems or string-overlap metrics, limiting their effectiveness on open-ended problems.Source: §1 · p. 2
  • RL post-training improves the SFT model by 4.1% and surpasses the base by 5.7%, with gains up to 14.6% on videos longer than 10 minutes and 4.8% on shorter videos.The authors describe these results as evidence of SAGE’s effectiveness on any-horizon video reasoning.Source: §1 · pp. 2–3

2. Related Work

Prior long-video agents commonly rely on temporal grounding, while video-reasoning RL methods use rewards that the authors consider suboptimal for open-ended problems. SAGE broadens tool use and applies an LLM judge to open-ended reward evaluation.

2.1. Long Video Reasoning Agents

  • SAGE combines temporal grounding with web search and speech transcription, and localizes events one short subclip at a time using coarse boundaries from SAGE-MM.The authors describe this as moving away from over-reliance on temporal grounding and as a more efficient approach than grounding over the entire video.Source: §2.1 · p. 3

2.2. Reinforcement Learning for Video Reasoning

  • Prior video-reasoning RL methods use option-matching or ROUGE rewards, while SAGE-MM uses GRPO with an LLM judge for open-ended answer rewards.The related-work passage notes that a concurrent method also uses LLM-as-a-judge rewards but supports only a crop-video tool call.Source: §2.2 · p. 4

3. Method

SAGE targets entertainment-video questions with an orchestrator that can answer directly or continue through tool calls. Its method combines a two-stage system, synthetic training data, and RL post-training for SAGE-MM.

3.1. System Design

  • SAGE-MM first builds video context and selects whether to answer or call a tool, then iteratively uses prior tool results for up to ten steps.The system takes sampled frames, video metadata, tool definitions, and the user query; segment-level event grounding is limited to a maximum duration of ten minutes.Source: §3.1 · p. 4

3.2. Synthetic Data Generation

  • Gemini-2.5-Flash generates 10–20 Q&A pairs per video, using a parsed-percentage field to span videos longer than five minutes; SAGE with Gemini generates four tool-call trajectories per question for cold-start SFT.The Q&A pairs support training and evaluation, while input-action pairs from unique trajectories form the cold-start dataset.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3. RL Post Training

  • The training data contains over 99k questions from more than 6600 videos on popular YouTube channels.These are the reported training-data statistics in Table 2.Source: Table 2 · p. 5
  • GRPO assigns each action a shared trajectory reward combining per-step scores with GPT-4o’s final-answer correctness verdict; early RL uses a six-step cap for stability.The reward design also discourages wrong answers after tool calls and gives a higher reward for correct answers using visual tools.Source: §3.3 · pp. 5–6
  • SAGE-Bench contains 1744 manually verified samples spanning diverse durations, emphasizing questions that require visual information.The benchmark is described as an evaluation set for entertainment-video reasoning.Source: Table 3 · p. 6

4. Experiments

Experiments evaluate SAGE’s implementation, benchmark, performance, and ablations. SAGE improves over baselines on open-ended tasks and longer videos, while its tool use and runtime results characterize the trade-offs of the system.

4.1. Implementation Details

  • The experiments train SAGE-MM using cold-start SFT and RL post-training across Molmo-8B, Qwen2.5-VL-7B-Instruct, and Qwen3-VL-4B/8B-Instruct backbones.The default ablation model is Qwen3-VL-8B-Instruct-based SAGE-MM.Source: §4 · p. 6
  • Training uses 99.1k questions from 6,659 videos, 417.7k SFT state–action pairs, and 7.68k RL samples split evenly between tool-call and single-turn responses.The evaluation uses 128 sampled frames for DIRECT baselines, adds transcripts for fair comparison, and uses GPT-4o as judge by default.Source: §4.1 · p. 6

4.2. SAGE-Bench

  • SAGE-Bench targets open-ended entertainment-video questions, with a disjoint-from-training sample manually checked and fewer than 5% of samples edited.The benchmark’s videos may overlap with training videos.Source: §4.2 · pp. 7–8

4.3. Main Results

  • +4.8% overall, +6.1% on open-ended questions, and +7.0% on visual questions versus Qwen2.5-VL-7B-Instruct; SAGE-Flash also improves over SAGE.The authors report up to 3.2% improvement over API base models and a further 3.8% improvement for SAGE-Flash over SAGE.Source: Table 4 · p. 7, §4.3 · p. 8
  • On MINERVA, SAGE improves 2.6% over the base model on videos longer than 600 seconds; SAGE-Flash outperforms baselines on Video-MMMU.On perception-centric Video-MME, disabling ground-event and extract-video-parts tools recovers competitive performance.Source: Table 5 · p. 7, Table 6 · p. 7, §4.3 · p. 8

4.4. Ablations

  • SAGE gains 8.2% in the 600–1200-second bucket, while SAGE-Flash gains 14.6%; both show larger gains on longer videos.The table caption also reports significant improvements for duration buckets above 600 seconds, with greater gains using Gemini-2.5-Flash as a tool.Source: Table 8 · p. 8, §4.4 · p. 9
  • RL refines the SFT model’s tool overcalling while improving single-turn and multi-turn accuracy, supporting SAGE’s any-horizon behavior.Removing transcription, video-part extraction, or analysis tools causes the largest performance declines; removing ground-event causes only a minor drop.Source: Table 10 · p. 8, Table 9 · p. 8, §4.4 · p. 9
  • SAGE’s runtime is comparable to Qwen3-VL-8B-Instruct with 512 frames, about one second slower than other thinking DIRECT baselines, and almost three times quicker than VideoMind.The authors attribute slower AGENT baselines to preprocessing and repeated model calls.Source: §4.4 · p. 9, Table 11 · p. 9

5. Conclusion

The conclusion presents SAGE as an any-horizon system for long-video reasoning and reports that its system design and RL post-training recipe improve performance, particularly on videos longer than ten minutes.

5. Conclusion

  • SAGE is an any-horizon reasoning system designed for long-video reasoning.The authors position the work as a proof-of-concept for practical AGENT systems beyond purely DIRECT approaches.Source: §5 · p. 9
  • The authors report that SAGE’s system design and RL post-training recipe enable any-horizon reasoning, with considerable gains on videos longer than 10 minutes.The conclusion also describes a cost-effective synthetic data pipeline for training and evaluation of open-ended entertainment-video queries.Source: §5 · p. 9
  • Future work includes broader-domain training data, agent-centric policy optimization, and systems that select appropriate tools or synthesize new ones.Source: §5 · p. 9

Appendix

The appendix outlines additional ablations and qualitative materials; it also reports that video access is critical for SAGE-Bench performance.

Appendix

  • Video access is critical for good performance on SAGE-Bench.Source: Table 12 · p. 14
  • The appendix covers video-input, cold-start SFT, and evaluation Nmax ablations, plus qualitative examples and prompt materials.It also includes QnA-generation examples and comparisons with existing benchmarks.Source: Appendix · p. 14
  • Unless otherwise noted, appendix experiments use Qwen3-VL-8B-Instruct-based SAGE-MM after RL on SAGE-Bench.Source: Appendix · p. 14

6. Additional Ablations

The ablations examine video-input dependence, inference mode, SFT, and how SAGE adjusts its reasoning horizon. They also report tool-wise performance and qualitative failure patterns.

Importance of the Video Input

  • Without video, SAGE-Bench performance drops by 27%, similar to the drop for the base Qwen3-VL-8B-Instruct model, providing no evidence of memorization.The questions are disjoint from training, although some benchmark videos overlap with training videos.Source: §6 · p. 14

Importance of SFT

  • For Qwen3-VL-4B-Instruct, SFT is essential before RL; for the 8B model, RL without SFT collapses to single-turn reasoning.The authors hypothesize that the base 8B model's answer-first training objective makes SFT necessary to incentivize any-horizon reasoning, while noting that extensive RL tuning might overcome this.Source: §6 · p. 14, Table 14 · p. 15

Eval Mode

  • AGENT inference outperforms DIRECT inference for the trained SAGE-MM.The authors note that Qwen3-VL performs better than Molmo2 in DIRECT mode, possibly because the model families differ in learning information directly despite AGENT-paradigm training.Source: Table 13 · p. 14, §6 · p. 14

#Turns v/s Video Duration

  • Average reasoning turns increase gradually with video duration, with shorter videos eliciting shorter trajectories and longer videos eliciting longer ones.The authors interpret this pattern as SAGE adapting its trajectory length to the input's temporal horizon.Source: §6 · p. 14, Table 15 · p. 15

Effect of Nmax

  • Nmax = 11 achieves high accuracy with few unanswered samples, and larger limits add only minimal gains.The authors conclude their RL recipe enforces answer prediction within an 11-step reasoning horizon.Source: §6 · pp. 14–15, Table 16 · p. 15

Variance on SAGE-Bench

  • Across five runs at temperature 1.0, Qwen3-VL-8B-Instruct has a SAGE-Bench standard deviation of 0.22.The authors interpret the low variance as indicating statistically significant performance improvements.Source: §6 · p. 15, Table 17 · p. 15

Per-Tool Accuracy

  • Among single-tool settings, transcribe-speech performs best, while extract-video-parts and ground-event perform worst.The authors attribute the latter tools' weaker performance to their dependence on tools such as analyze and extract-video-parts for local segment processing.Source: §6 · p. 15, Table 18 · p. 15

Qualitative Failure Analysis

  • Observed failures include tool retries that consume reasoning steps, missed segments from inaccurate temporal grounding, and incorrect answers from misleading transcript or web information.Source: §6 · p. 15

7. SAGE-Bench

SAGE-Bench is designed to evaluate both practical entertainment-video questions and diagnostic capabilities. It uses open-ended questions to reflect practical user situations.

High-Quality Open-Ended Questions

  • Unlike the multiple-choice-only popular benchmarks described, SAGE-Bench uses open-ended questions with an unbounded answer space.The paper presents this format as more closely aligned with practical user situations.Source: §7 · p. 15

Dual Focus on Diagnostic and Practical Evaluation

  • Figure 5 illustrates SAGE-Bench examples spanning practical scenarios and diagnostic cases, alongside representative examples from other benchmarks.Source: Figure 5 · p. 16
  • SAGE-Bench includes diagnostic questions about fundamental capabilities and practical questions users may ask while watching entertainment videos.The benchmark is intended to assess both technical proficiency and utility in practical scenarios.Source: §7 · pp. 15–16

8. System Prompts

The system-prompt appendix identifies prompts for generating QnA pairs, LLM-judge evaluation, and SAGE's two reasoning stages. It also lists prompts for ground-event, step-reward computation, and DIRECT-baseline evaluation.

System Prompts

  • The listed prompts cover QnA-pair generation and LLM-judge evaluation, as well as SAGE's Stage-1 Context VLM and Stage-2 Iterative Reasoner.Source: §8 · p. 16
  • Additional prompts are provided for the ground-event tool, reasonable-tool step-reward computation, and DIRECT-baseline evaluation.Source: §8 · p. 16

9. Qualitative Examples

The qualitative materials show generated QnA samples, describe the roles of SAGE's two reasoning stages, and present trajectories with different turn counts. They also include the ground-event tool's timestamp instruction.

SAGE Any-Horizon Reasoning Trajectories

  • The examples present SAGE trajectories using five turns, two turns, and a single turn.The paper offers these examples to demonstrate SAGE's any-horizon reasoning abilities.Source: §9 · p. 16

SAGE Stage-1 and Stage-2 Prompts

  • The Context VLM analyzes the video and query, recommends tools, and suggests arguments; the Iterative Reasoner checks whether available context suffices and recommends further tools if needed.Source: §9 · pp. 18–19

Ground-Event Tool

  • The ground-event instruction asks for start and end timestamps aligned to the original video, or null timestamps if the event does not occur.Source: §9 · p. 20

QnA Pairs

  • The QnA-pair samples are described as varying in difficulty and question type, including open-ended and multiple-choice questions, and covering the entire video.Source: Figure 13 · p. 21, Figure 14 · p. 22
Loading 2512.13874v2…