Source-linked AI summary
LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoning
Shenghao Fu, Qize Yang, Yuan-Ming Li, Xihan Wei, Xiaohua Xie, Wei-Shi Zheng
TL;DR
Long-video LVLMs struggle to balance temporal coverage and spatial detail under a limited context length. LOVE-R1 uses slow-fast adaptive sampling and multi-step reasoning to zoom into relevant clips, and it outperforms Qwen2.5-VL by an average of 3.1% points across four benchmarks. The authors also report limited 16k context as a boundary for further scaling.
Problem
Uniform frame sampling forces long-video LVLMs to trade off temporal clues against spatial details under limited context length.
Method
LOVE-R1 densely samples a video at low resolution, then adaptively zooms into relevant clips at high resolution through multi-step reasoning and decoupled reinforcement finetuning.
Results
LOVE-R1 outperforms Qwen2.5-VL by an average of 3.1% points across 4 common long video understanding benchmarks.
Takeaways & Limitations
The slow-fast adaptive frame-sampling mechanism achieves a trade-off between sampling density and spatial details while supporting state-of-the-art long-video benchmark performance.
Takeaways & Limitations
The model is limited to a 16k context, and extending context length is left as future work.
Abstract
from arXiv · showhide
Long video understanding is still challenging for recent Large Video-Language Models (LVLMs) due to the conflict between long-form temporal understanding and detailed spatial perception. LVLMs with a uniform frame sampling mechanism, which samples frames with an equal frame size and fixed sampling rate, inevitably sacrifice either temporal clues or spatial details, resulting in suboptimal solutions. To mitigate this dilemma, we propose LOVE-R1, a model that can adaptively zoom in on a video clip. The model is first provided with densely sampled frames but in a small resolution. If some spatial details are needed, the model can zoom in on a clip of interest with a large frame resolution based on its reasoning until key visual information is obtained. The whole process is implemented as a multi-step reasoning process. To train the reasoning ability, we first finetune the model on our collected 38k high-quality CoT data and enhance it with decoupled reinforcement finetuning. As outcome rewards can not provide fine-grained process supervision, we decouple multi-step reasoning into multiple single-step reasoning and optimize the internal zoom-in ability explicitly. Experiments on long video understanding benchmarks show that our model with the slow-fast adaptive frame sampling mechanism achieves a great trade-off between sampling density and frame resolutions, and LOVE-R1 outperforms our baseline Qwen2.5-VL by an average of 3.1% points across 4 common long video understanding benchmarks.
1 INTRODUCTION
LOVE-R1 addresses the temporal–spatial trade-off in long video understanding by adaptively zooming into query-relevant clips through multi-step reasoning. It combines dense low-resolution global sampling with selective high-resolution frames and reports improvements over Qwen2.5-VL across long-video benchmarks.
- Motivation and approach: Uniform frame sampling forces a trade-off between temporal clues and spatial details because context length limits the number and resolution of frames.
- Motivation and approach: More than 75% Pass@N accuracy is achievable with 32 random frames on most benchmarks, while about 90% of query-induced attention concentrates on 5% of frames.
- Motivation and approach: LOVE-R1 adaptively zooms into relevant video clips, combining global temporal coverage with high-resolution spatial detail through multi-step reasoning.The model decides whether visual information is sufficient, selects a time span when needed, and then answers.
- Results: 48.2% on LVBench, 60.1% on LongVideoBench, and 66.2% on VideoMME were reported for LOVE-R1.
2 RELATED WORK
Prior work compresses or selects video information and increasingly applies reasoning to multimodal inputs. LOVE-R1 instead combines dynamic video processing with a slow-fast template and decoupled process rewards.
- Video processing: Long-video LVLMs commonly use fixed-rate, fixed-resolution sampling, which can exceed the context budget as videos become longer.
- Video processing: Video methods reduce visual-token cost through token compression or keyframe selection, but these approaches prune or reorganize information at different granularities.
- Multimodal reasoning: Reasoning models decompose difficult problems into solvable subproblems, and early video-reasoning work indicates that visual information in thought can improve perception and reasoning.
- Positioning: Unlike concurrent VITAL and Video-MTR, LOVE-R1 uses a slow-fast template and decoupled reinforcement finetuning to balance temporal density, resolution, and process supervision.
3 A DYNAMIC FRAME PROCESSING MECHANISM
The dynamic frame-processing mechanism uses a low-resolution, densely sampled fast video for global temporal coverage and appends high-resolution slow clips selected by the model. This preserves temporal context while adding query-relevant spatial detail.
- Design motivation: Uniform sampling exceeds context limits on long videos, forcing a loss of either temporal or spatial information when sampling rate or resolution is reduced.
- Slow-fast mechanism: The fast video densely samples the full video at high frame rate and small resolution, while selected clips are sampled at high resolution with fewer frames.
- Slow-fast mechanism: LOVE-R1 appends slow videos after the fast video and preserves the corresponding fast segments rather than replacing them.
- Slow-fast mechanism: The template lists the full fast video followed by ordered zoom-in clips, each represented as a separate slow video segment.
- Template choice: Template (c) was adopted because appending slow videos aligns with the pretraining template and requires little finetuning data.
- Adaptive selection: The model uses reasoning to decide whether information is sufficient and, if not, selects the relevant clip to zoom in on.
4 A THREE-STAGE POST-TRAINING RECIPE
LOVE-R1 is trained through slow-fast template finetuning, CoT cold start, and decoupled reinforcement finetuning. The recipe teaches adaptive zoom-in behavior and explicitly supervises clip selection when outcome rewards are insufficient.
- Three-stage recipe: The three-stage recipe first adapts the model to the slow-fast template, then teaches reasoning from 38k CoT examples, and finally reinforces zoom-in behavior.
- Stage 1: slow-fast template finetuning: Template finetuning lets the model adopt multiple segments with different frame rates, resolutions, and timespans without sacrificing performance.
- Stage 2: CoT cold start: The 38k CoT dataset is built from grounded video question-answering data with annotated timespans, enabling filtering of incorrect zoom-in reasoning.
- Stage 3: decoupled reinforcement finetuning: Outcome rewards can encourage incorrect zoom-in clips when the final answer is correct, so decoupled reinforcement finetuning explicitly optimizes single-step zoom-in behavior.
- Stage 3: decoupled reinforcement finetuning: For general questions, standard GRPO optimizes multi-step CoTs using shared answer rewards, while timespan-supervised data explicitly rewards zoom-in overlap.
5 EXPERIMENT
LOVE-R1 is evaluated through benchmark comparisons and ablations of its adaptive zoom-in mechanism, training recipe, and multi-step inference. The experiments show improved long-video performance and a balance between temporal coverage and spatial detail.
- 5.3 ABLATION STUDIES: Coupling multi-step and single-step optimization reaches 66.2% on VideoMME, outperforming standard GRPO by 0.5%.The decoupled reinforcement finetuning explicitly optimizes zoom-in chain-of-thoughts because outcome rewards provide insufficient process supervision.
- 5.3 ABLATION STUDIES: CoT cold-start training raises LongVideoBench and LVBench performance by 4.1% and 1.5% while preserving other benchmark results.After decoupled reinforcement finetuning, the Stage 3 model achieves the highest overall performance.
- 5.3 ABLATION STUDIES: Adaptive zoom-in achieves the highest performance against no-zoom, uniform-zoom, and random-zoom baselines, indicating question-related clip selection.The multi-step model can further correct an initially incorrect clip choice by sampling other clips.
- 5.3 ABLATION STUDIES: Performance improves as maximum inference iterations increase from one to three, then saturates after three steps.One-step inference reaches 60.9%, while additional steps enable zooming into relevant clips and correcting mistaken selections.
- 5.2 MAIN RESULTS: Adaptive zoom-in balances sampling density and spatial details better than fixed-frame reasoning models across long and short video settings.Fixed slow sampling preserves spatial details but misses temporal clues, whereas fixed fast sampling preserves temporal clues but loses spatial detail.
6 CONCLUSION
The paper formulates long video understanding as multi-step reasoning with adaptive zoom-in and trains explicit zoom-in supervision through decoupled reinforcement finetuning. LOVE-R1 achieves strong benchmark performance, while data quality and a 16k context limit remain constraints.
- 6 CONCLUSION: LOVE-R1 combines decision, zoom-in, and answering abilities to obtain spatial details from selected clips before producing a final answer.Its slow-fast adaptive frame sampling mechanism balances sampling density and spatial details.
- 6 CONCLUSION: Decoupled reinforcement finetuning provides explicit optimization for internal zoom-in ability beyond outcome-reward training.The method decomposes multi-step reasoning into multiple single-step reasoning processes.
- 6 CONCLUSION: Performance is constrained by the quality of existing long video understanding training data and by the model's 16k context limit.The paper identifies larger context lengths as future work for processing more frames and reasoning steps.
A MORE EXPERIMENT RESULTS
LOVE-R1 is evaluated on short-video benchmarks and achieves competitive performance there.
- LOVE-R1 achieves competitive short-video understanding performance on MVBench and the short part of Video-MME.
B.1 SOURCE DATA SELECTION
The source data combines grounded video question-answering datasets with annotated local timespans and global questions selected for CoT diversity.
- NExT-GQA and CG-Bench provide questions annotated with related timespans for grounded video question answering.NExT-GQA contains videos within 3 minutes, whereas CG-Bench contains videos ranging from 10 to 60 minutes.
- Additional sources include 2–3-minute LLaVA-Video-178k videos with pseudo timespans and FineVideo global questions targeting main-purpose, main-character, and main-message information.
B.2 COT DATA CONSTRUCTION PIPELINE
The CoT construction pipeline uses separate prompts, strong-model annotation, filtering, and standardized formatting to produce training data for zoom-in and answer reasoning.
- The pipeline collects separate zoom-in and answer CoTs for grounded datasets, while FineVideo contributes answer CoTs only.
- Question cleaning removes CG-Bench examples with irrelevant timespans or questions unanswerable from ground-truth clips.
- Long CG-Bench videos are divided into non-overlapping 10-second clips and captioned with Qwen2.5-VL 7B for localization annotation.
- Gemini 2.5 pro annotates CoTs with different prompts, using raw videos for short-video annotation and captions or ground-truth clips for long-video annotation.
- The pipeline filters wrong answers, incorrect timespans with IoU < 0.1, repeated patterns, and undesired styles before standardizing time representations to seconds.The resulting dataset contains 38k CoTs.
- LLMs support the work by collecting the CoT dataset and improving the paper writing.
D MORE VISUALIZATION
Inference visualizations show LOVE-R1 selecting clips of interest for adaptive zoom-in, which enhances long-video understanding.
- LOVE-R1 selects clips of interest to zoom in on, enhancing long-video understanding in the visualized inference results.