Source-linked AI summary
Enhancing Localized Reasoning for Long Video Understanding via Efficient Segment-to-Video Supervision
Beibei Zhang, Chao Xu, Jun Lan, Zongyi Li, Lai Wei, Huijia Zhu, Tongwei Ren
TL;DR
Long-video understanding suffers when distracting context obscures localized evidence, while existing reasoning approaches bring substantial training and inference costs. S2V transfers short-segment VQAs to whole-video training using lightweight RL, improving LVU accuracy and efficiency across multiple benchmarks.
Problem
Existing LVU methods face distracting long-video context and substantial training or inference overheads.
Method
S2V generates VQAs from localized segments, transfers them to whole videos with segment timestamps, and trains using RL with a simple accuracy reward.
Results
Across multiple benchmarks, S2V consistently improves LVU performance and surpasses general MLLMs and reasoning-based methods in accuracy and training and inference efficiency.
Takeaways & Limitations
S2V enhances fine-grained evidence comprehension while using 10K VQA samples and single-pass inference with limited output tokens.
Takeaways & Limitations
S2V is not designed for exact evidence grounding and yields limited grounding improvement because it lacks exact localization annotations and a grounding-targeted objective.
Abstract
from arXiv · showhide
Though Multimodal Large Language Models (MLLMs) have shown impressive potential in video understanding, long video understanding (LVU) remains challenging since distracting noise in complex and lengthy contexts can obscure localized details, misleading MLLMs to produce incorrect answers. Recent works mitigate these issues by incentivizing deep reasoning to include relevant evidence. However, these methods have two main problems: First, the reinforcement fine-tuning framework (RFT) they leveraged incurs substantial training overheads, including high annotation costs and complicated reward designs. Second, the self-reflective and iterative-perception mechanism in some methods causes lengthy outputs and high inference latency. To alleviate these problems, we propose a novel Segment-to-Video Supervision} method (S2V) to efficiently enhance fine-grained reasoning in LVU. Specifically, we generate question answer pairs (VQA) based on localized segments, and then transfer these segment-based VQA back to the whole video for training. Due to focusing on short segments, segment-based VQA can naturally notice details which tend to be overlooked from a whole-video perspective. Training on such data can enforce MLLMs to correctly associate fine-grained details with QA while avoiding distracting noise in the whole video. The S2V training involves just reinforcement learning (RL) with a simple accuracy reward based on only 10K VQA samples and the resulting S2V model predicts answer using a single forward pass with limited output tokens. Experimental results demonstrate that S2V can consistently improve LVU performance across multiple LVU benchmarks, outperforming both general MLLMs and reasoning-based methods not only in LVU accuracy but also in training and inference efficiency.
Introduction
Long-video understanding is hindered by distracting context and costly reasoning pipelines. S2V addresses both issues by transferring localized segment supervision to whole-video training with lightweight reinforcement learning and inference.
- Distracting noise in long-form contexts can prevent MLLMs from identifying useful details for LVU.
- Retrieval-based solutions add training and inference overhead, while reasoning-based approaches can require costly annotations, complex rewards, and lengthy outputs.
- S2V generates VQAs from short segments and transfers them to whole videos using segment timestamps, associating localized details with answers.
- S2V uses only RL with a simple accuracy reward, 10K VQA samples, and a single forward pass with limited output tokens.
- Across multiple LVU benchmarks, S2V improves accuracy over general MLLMs and LVU-tailored methods while reducing training and inference costs.
Related Work
Prior LVU methods use retrieval or evidence-focused reasoning to combat distracting long-video context, but these strategies impose computational, annotation, or inference costs. S2V instead uses compact segment-to-video supervision.
- General MLLMs struggle to capture detailed evidence because long-form videos contain abundant visual noise.
- Retrieval-based methods retrieve query-relevant frames but incur computational and storage burdens from massive query-frame retrieval.
- Jointly training retrievers and answerers can create considerable training overheads.
- Reasoning-based methods use SFT and RL, requiring costly chain-of-thought annotations, complicated rewards, or lengthy iterative outputs.
- S2V uses only 10K VQA samples, a simple accuracy reward, and one forward pass while surpassing reasoning-based methods in accuracy and efficiency.
Method
S2V constructs and filters segment-focused VQAs, transfers them to whole videos with timestamp guidance, and trains MLLMs using a simple accuracy-based RL objective. The resulting data emphasizes localized evidence while the method remains efficient, though it is not designed for exact grounding.
- Video Collection: Videos are selected from YT-Temporal-180M and segmented at visual scene transitions to preserve continuity.
- Segment-based and Video-based VQA Generation: MLLMs generate single- and multi-segment VQAs, while verification removes inconsistent answers and irrelevant segment associations.
- Segment-based and Video-based VQA Generation: Segment timestamps convert segment VQAs into unambiguous whole-video VQAs by directing attention to the corresponding segments.
- Data Filtering: Difficulty filtering produces 1,000 simple, 3,000 medium, and 6,000 hard samples in S2V-10K.
- Data Statistics and Quality: S2V-10K contains more than one segment per sample on average and spans diverse video durations, topics, and question types.
- Optimization Objective: The RL objective assigns reward 1 to correct answers and 0 to incorrect predictions using whole video, question, timestamp instruction, and answer tuples.
- Why Does S2V Work?: Short-segment VQAs expose fine-grained local evidence, helping models distinguish useful details from distracting noise.
- S2V Training: S2V is not designed for evidence grounding, and its timestamps are training-time privileged information rather than exact grounding annotations.
Experiments
Across multiple LVU benchmarks, S2V improves accuracy over general and LVU-tailored methods while reducing training and inference costs. Experiments also show stronger localized-evidence use, robustness across frame counts, and benefits from larger models and balanced training difficulty.
- Comparison with General MLLMs: S2V consistently improves LVU performance across LongVideoBench, Video-MME, and MLVU, including stronger results on medium- and long-video subsets.It also performs strongly on MLVU’s needle-in-a-haystack and counting subsets.
- Latency Comparison: S2V achieves a better accuracy-latency tradeoff than general MLLM and reasoning-based baselines through a single forward pass rather than iterative perception.Even S2V-4B achieves high LVU accuracy with low inference cost.
- Fine-grained Reasoning Enhancement (RQ3): S2V models rely more strongly on localized evidence because removing corresponding segments causes larger accuracy drops than for their base models.Segment-view accuracy exceeds whole-video accuracy, while S2V training narrows the gap between them.
- Ablation Study (RQ4 & RQ5): Across all simple-medium-hard ratios, S2V improves over the base MLLM, while a balanced 1:3:6 ratio avoids the weaknesses of only simple or only hard samples.Only simple samples converge rapidly but yield sub-optimal improvement; only hard samples create instability and convergence challenges.
- Ablation Study (RQ4 & RQ5): S2V-8B consistently outperforms S2V-4B across data scales, and additional data further improves S2V-8B performance.S2V-4B performance slightly drops as training data increases.
Conclusion
The paper proposes Segment-to-Video Supervision to address training and inference overheads in reasoning-based LVU methods. S2V transfers segment-generated VQAs to whole videos, improving LVU performance and efficiency across benchmarks.
- Conclusion: S2V generates VQAs from short segments and transfers them back to whole videos for supervision.This trains models to associate fine-grained evidence with answers while avoiding distracting whole-video noise.
- Conclusion: S2V uses reinforcement learning with a simple accuracy reward and only 10K VQA samples.The training design avoids the larger annotation and reward complexity associated with reasoning-based approaches.
- Conclusion: S2V predicts answers with a single forward pass and limited output tokens.This design targets lower inference latency than iterative reasoning procedures.
- Conclusion: Across multiple benchmarks, S2V consistently improves LVU performance while outperforming general MLLMs and reasoning-based methods in accuracy and efficiency.The reported efficiency dimensions are training and inference efficiency.
Appendix
The supplementary material provides benchmark details, training specifications, grounding experiments, an RL-versus-RFT ablation, and case studies of data construction and reasoning outputs.
- Appendix: The supplement includes details of the evaluation benchmarks and S2V training procedure.
- Appendix: It reports experiments on video grounding and an ablation comparing RL with RFT for S2V training.
- Appendix: The case studies cover S2V-10K samples, failed data-construction examples, and S2V reasoning outputs.
Evaluation Benchmark Details
The evaluation uses LongVideoBench, Video-MME, and MLVU, covering diverse domains, durations, and long-video understanding question types.
- LongVideoBench: LongVideoBench contains 6,678 multiple-choice questions on 3,763 diverse videos spanning four duration groups.Its questions cover 17 fine-grained categories with human-annotated choices, and the paper evaluates on its validation set.
- Video-MME: Video-MME contains 2,700 multiple-choice questions on 900 videos across six domains and short, medium, and long durations.The reported median lengths are 26s, 164.7s, and 890.7s for the three duration groups.
- MLVU: MLVU contains 3,102 questions on 1,730 videos across nine categories designed specifically for long video understanding.Videos range from 3 minutes to 2 hours, and the paper evaluates its multiple-choice set, including Plot-QA, Needle, Ego, and Count subsets.
S2V Training Details
S2V training uses LoRA with GRPO under specified hardware, optimization, rollout, and image-resolution settings, while the supplement documents prompts for data synthesis.
- S2V Training Details: S2V uses LoRA for GRPO training on 8 80G A100 GPUs for 1 epoch with AdamW.The configuration uses rank 32, rollout group 8, temperature 1.0, batch size 64, and learning rate 1 × 10^-5.
- S2V Training Details: Training and inference use maximum image pixels of 50,176 and 174,080, respectively.
- S2V Training Details: The supplement provides prompts for segment-based VQA generation and segment-question relevance checking in S2V-10K synthesis.These prompts are presented to improve reproducibility and transparency.
Video Grounding Evaluation
S2V’s grounding capability is evaluated on TVGBench, with results reported in Table 4. The gains are consistent but limited because S2V lacks exact evidence positions and grounding-specific optimization.
- TVGBench evaluates S2V’s temporal video grounding capability.The benchmark contains 800 balanced-duration instances across 11 semantic categories and three major types: human, object, and environment.
- Table 4 compares S2V with general MLLMs and TVG-tailored methods on TVGBench.
- S2V consistently improves grounding capability, but the performance gain is not particularly significant.
- S2V’s grounding gains are limited because its annotations lack exact evidence positions and its training has no grounding-related optimization objective.
RL vs. SFT
S2V uses only reinforcement learning because its MCQ supervision provides little semantic information for supervised fine-tuning. On the same S2V-10K data, RL substantially outperforms SFT for LVU performance.
- S2V’s MCQ annotations provide only a correct option label, such as A, B, or C.
- S2V therefore uses only RL, because single-letter MCQ signals are unsuitable for SFT.
- On the same S2V-10K data, RL substantially outperforms SFT, while SFT degrades LVU performance versus the base MLLM.
Case Study
The case studies illustrate how S2V data are generated, where construction can fail, and how S2V improves localized evidence understanding in long videos.
- S2V-10K Sample: The generation prompt produces a multi-choice question-answer-distractor triplet from segment frames and a question template.
- S2V-10K Sample: S2V-10K converts segment-based VQA into video-based VQA by incorporating the corresponding segment location.
- Failed Data Construction Examples: The automated pipeline reaches a 93% human pass rate, but failures arise from object-recognition errors, confusing distractors, and segment ambiguity.
- S2V Reasoning Outputs: S2V captures localized details overlooked by the base model, such as a third jet-skiing scene hidden among surrounding ocean scenes.
- S2V Reasoning Outputs: S2V correctly associates visual evidence with answers, including recognizing toy cows as milking-cow scenes and comprehending temporal relations.
- S2V-10K Sample: The segment-question relevance check verifies whether every segment in a multi-segment sample is relevant to its multi-choice question.