PaperlayerView exact revision on arXiv

Source-linked AI summary

RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding

Shanqing Xu, Meng Luo, Mengchen Qian, Yuhui Gao, Siyue Peng, Xiaohan Zhong, Xiaojin Zhang, Zhongyu Wei, Wei Chen, Xiang Bai

arXiv:2608.29958v1cs.CV

TL;DR

Long-video LVLMs face a fixed visual-token budget, while relevance-based frame selection can overlook temporally distributed evidence around an event. RIDGE reads frame-query similarity as a temporal signal, segments it into structural regions, and applies question-aware, region-specific selection; across four benchmarks and three backbones, it delivers consistent gains over uniform sampling and strong keyframe-selection baselines.

  • Problem

    Existing query-aware selectors often treat frame-query similarity mainly as values to rank or sample, which can miss buildup, aftermath, transitions, and contextual evidence around relevant events.

  • Method

    RIDGE uses local slope and curvature on precomputed frame-query scores to segment structural regions, allocate a question-aware frame budget, and select frames with region-specific rules.

  • Results

    Across four long-video benchmarks and three LVLM backbones, RIDGE delivers consistent gains over uniform sampling and strong keyframe-selection baselines, with especially clear benefits for sparse or temporally distributed evidence.

  • Takeaways & Limitations

    RIDGE provides a lightweight, training-free, backbone-agnostic post-processing approach for preserving event cores and surrounding temporal evidence under a fixed frame budget.

  • Takeaways & Limitations

    RIDGE depends on meaningful frame-query scores, a capable downstream LVLM, and a predefined candidate frame sequence, leaving missed brief events, fine-grained OCR, and small objects inaccessible.

Abstract

from arXiv · show

Long videos contain far more visual content than Large Vision-Language Models (LVLMs) can process under a fixed visual-token budget, making frame selection essential. Existing query-aware selectors usually estimate frame-query relevance and build a compact subset from high-scoring frames. Although their mechanisms differ, the similarity sequence is still often treated primarily as values to rank or sample from, rather than as an ordered signal whose shape reflects how query-relevant evidence emerges, peaks, and fades over time. This can obscure frames that explain, contextualize, or follow an event, because such evidence may lie on the rising or falling sides of a nearby relevance peak and receive lower absolute scores. We propose RIDGE, a frame selection framework that reads the frame-query similarity curve as a temporal signal. By using local changes and curvature, RIDGE partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget. It is a lightweight post-processing step on precomputed frame-query scores and requires neither training nor iterative LVLM calls. Across four long-video benchmarks and three backbones, RIDGE achieves the best performance in most settings and remains competitive in the others.

1 Introduction

Long-video frame selection must preserve query-relevant evidence under a fixed visual-token budget. RIDGE treats relevance scores as an ordered temporal signal and selects across event cores, buildup, aftermath, transitions, and context.

  • A one-hour video sampled at 1 FPS can produce thousands of frames and millions of visual tokens, making full-input processing costly.
  • Selecting only the highest-scoring frames can miss buildup and aftermath evidence needed to distinguish an event.
  • Existing query-aware selectors largely treat similarity magnitude as the primary signal, using temporal structure mainly for coverage or search.
  • RIDGE interprets the frame-query similarity sequence as a query-conditioned temporal signal whose local shape reveals approach, event core, departure, transitions, and context.
  • RIDGE segments precomputed scores into structural regions, allocates the frame budget using question preferences, and applies region-matched selection without training or iterative LVLM calls.
  • Across four long-video benchmarks and three LVLM backbones, RIDGE reports consistent gains over uniform sampling and competitive or superior performance to stronger baselines.

2 Related Work

Prior long-video systems improve visual representation or frame selection through coverage, diversity, adaptive search, and refinement. RIDGE addresses the comparatively underexplored temporal shape of the relevance signal.

  • Long-video LVLM research improves visual evidence representation through dynamic resolution, time-aware encoding, unified image-video training, and larger visual backbones.
  • Query-agnostic sampling provides broad temporal coverage but cannot adapt to evidence required by a particular question.
  • Query-aware keyframe methods construct relevance signals and extend top-k selection with temporal structure, diversity, adaptive search, or model-guided refinement.
  • From RIDGE’s perspective, prior work leaves the internal temporal shape of the relevance signal underexplored.

3 Method

RIDGE converts query-frame relevance scores into temporal evidence roles, allocates a question-dependent frame budget, and selects representative frames using role-specific criteria. Its segmentation uses normalized, smoothed scores together with slope and curvature to recover event structure and temporal change.

  • 3.1 Overview: A lightweight vision-language scorer produces a query-frame relevance sequence for candidate video frames and the question.
  • 3.1 Overview: RIDGE maps the relevance curve into five roles: PEAK for event cores, RISING for buildup, FALLING for aftermath, BOUNDARY for abrupt changes, and BACKGROUND for context.
  • 3.2 Temporal Region Segmentation: RIDGE normalizes and Gaussian-smooths scores, then computes first- and second-order differences for temporal analysis.
  • 3.2 Temporal Region Segmentation: Slope indicates whether relevance increases or decreases, while curvature helps distinguish sharp peaks from temporally extended ones.
  • 3.2 Temporal Region Segmentation: Local maxima receive curvature-adaptive PEAK windows, while adjacent positive- and negative-slope spans become RISING and FALLING regions.
  • 3.3 Question-aware Budget Allocation: Question-derived coverage intent controls how broadly rising and falling regions extend around an event core.
  • 3.4 Role-specific Frame Selection: RIDGE assigns residual high-|s′| frames to BOUNDARY and remaining frames to BACKGROUND, then ranks or covers frames according to each region’s temporal function.
  • 3.3 Question-aware Budget Allocation: The method allocates the frame budget across regions according to question-dependent evidence preferences, with capacity caps and redistribution of unused budget.

4 Experiments

RIDGE is evaluated across four long-video benchmarks, multiple LVLM backbones, matched scoring controls, robustness settings, and component analyses. Results show consistent gains over uniform sampling, with especially strong improvements on longer and multi-event reasoning cases.

  • Experimental Setup: RIDGE is evaluated on Video-MME, LongVideoBench, MLVU, and LVBench using three LVLM backbones.The study reports accuracy under a unified evaluation pipeline and compares RIDGE with uniform, adaptive, score-based, and diversity-aware selectors.
  • Main Comparison: 12.4 percentage points is RIDGE’s improvement over uniform sampling on LVBench with Qwen2.5-VL-7B.The corresponding improvements are 2.5 points on Video-MME, 6.5 on LVB, and 9.9 on MLVU.
  • Matched-Scorer Comparison: 3.5 percentage points is RIDGE’s four-benchmark average improvement over BOLT with CLIP scores.With SigLIP scores, RIDGE improves the average over MDP3 by 3.8 points and is higher in seven of eight matched comparisons.
  • Question-aware Weighting: 61.1 is RIDGE’s four-benchmark average with fixed weights, exceeding FOCUS by 1.4 points and AKS by 1.9 points.Question-specific weights generated by Qwen3-8B raise the average to 62.4, while Qwen3-0.6B retains most of the benefit.
  • Robustness Analysis: RIDGE improves over uniform sampling across frame budgets, LVLM scales, scoring models, weight-generation LLMs, and smoothing settings.At N=64 the gain remains positive, and the 72B model still benefits by 6.3 percentage points; accuracy peaks at σ=2.0 and degrades gracefully outside it.
  • Ablation and Analysis: RIDGE improves substantially on LVBench information retrieval, entity recognition, reasoning, and temporal grounding, while gains are smaller on summarization.Across LongVideoBench duration buckets, gains are moderate for short videos and reach +8.33 percentage points on the 3600-second bucket.

5 Conclusion

RIDGE treats frame-query similarity as an ordered temporal signal and selects frames by structural region rather than score magnitude alone. Across four long-video benchmarks and three LVLM backbones, it consistently improves over uniform sampling and strong keyframe-selection baselines.

  • RIDGE reframes query-aware keyframe selection as reading a query-conditioned temporal signal whose local shape encodes different evidence roles.The framework distinguishes event cores from buildup, aftermath, transitions, and contextual evidence.
  • RIDGE segments the similarity signal into structural regions, allocates budget according to question-dependent evidence needs, and applies region-specific selection rules.
  • Across four long-video benchmarks and three LVLM backbones, RIDGE delivers consistent gains over uniform sampling and strong keyframe-selection baselines.Benefits are especially clear when evidence is sparse or temporally distributed.

Limitations

RIDGE’s scope is constrained by the temporal structure, scorer quality, downstream LVLM, and predefined candidate-frame sequence available to it. These boundaries limit its advantage for short or single-frame questions and can leave brief, fine-grained, or initially missed evidence inaccessible.

  • RIDGE may offer only modest advantages for short videos or questions answerable from a single salient frame.Such settings provide less temporal structure to recover.
  • RIDGE depends on meaningful frame-query score variation and on the downstream LVLM’s ability to interpret selected visual evidence.Better frame allocation alone cannot fully solve the question when either condition fails.
  • RIDGE selects from a predefined candidate frame sequence, so very brief events, fine-grained OCR, small objects, or initially missed evidence can remain inaccessible.The paper suggests denser candidate generation, stronger scorers, or joint scoring and reasoning as future directions.

C Compared Frame-selection Methods

Compared methods address query-aware frame selection through fixed coverage, adaptive temporal partitioning, stochastic relevance sampling, diversity-aware optimization, or budgeted exploration. Their shared goal is to retain relevant evidence while controlling temporal coverage, redundancy, or scoring uncertainty.

  • Uniform sampling: Uniform sampling places frames at evenly spaced temporal locations, preserving coarse coverage without a scoring model or additional selection algorithm.Its question-independent allocation can miss sparse relevant evidence and spend frames on uninformative spans.
  • AKS: AKS recursively splits or selects temporal bins using prompt–frame matching scores to balance relevance against temporal coverage.This contrasts top-score sampling’s narrow concentration with binned sampling’s stronger coverage.
  • BOLT: BOLT converts query–frame similarities into a probability distribution and uses inverse transform sampling, with sharpness controlling preference for high scores.
  • MDP3: MDP3 jointly models query relevance, diversity, and temporal sequentiality through a similarity matrix, determinantal point process, and dynamic programming.Its selection-size allocation is formulated as a Markov decision process.
  • FOCUS: FOCUS treats short temporal clips as arms in a budgeted combinatorial pure-exploration problem, prioritizing clips that are promising or uncertain.Its two-stage schedule combines coarse parallel exploration with batched optimistic exploration.

D Evaluation Benchmarks

The evaluation uses benchmarks spanning broad video analysis, long-context referring reasoning, multi-task long-video diagnosis, and extreme long-form understanding. Together, they cover varied durations, domains, task formats, and temporal reasoning demands.

  • Video-MME: Video-MME evaluates multimodal video analysis across 900 videos and 2,700 multiple-choice question–answer pairs spanning six visual domains and 30 categories.Videos range from 11 seconds to one hour; the experiments use the video-only setting without subtitles.
  • LongVideoBench: LongVideoBench evaluates long-context interleaved video–language understanding on 3,763 web videos with 6,678 human-annotated multiple-choice questions.Its referring-reasoning task requires retrieving and reasoning over relevant video contexts.
  • MLVU: MLVU diagnoses long-video understanding across genres, durations, and task formats, including topic reasoning, anomaly recognition, summarization, and several forms of question answering.Its videos span roughly three minutes to more than two hours and average about 12 minutes.
  • LVBench: LVBench targets extreme long-form understanding with 103 publicly sourced videos averaging more than one hour and 1,549 manually annotated question–answer pairs.Its questions probe key information retrieval, event understanding, entity recognition, and reasoning capabilities.

E Complexity and Runtime Analysis

RIDGE adds lightweight post-processing to precomputed similarity scores, with O(T log T) worst-case complexity and 8.1 ms of per-video overhead. It remains close to non-iterative selectors and far faster than LVLM-iterative methods.

  • Complexity: RIDGE runs in O(T log T) worst-case time, asymptotically comparable to a plain top-k selector.Its stages include O(T) region analysis followed by ranked selection and backfilling.
  • Runtime: 8.1 ms is RIDGE’s reported per-video post-processing time on LVB, excluding shared similarity computation.The optional Qwen3-8B weight call is performed once per input question and adds a small preprocessing cost.
  • Runtime: RIDGE remains close to lightweight non-iterative selectors and far faster than LVLM-iterative methods.The comparison excludes the shared cost of computing frame-query scores.

F Hyperparameter Sensitivity

RIDGE remains substantially above uniform sampling across tested hyperparameter configurations, indicating robustness rather than dependence on a narrow setting. Temporal-context parameters, especially k_s, produce the largest sensitivity, while safety clamps have minimal effect.

  • Robustness: RIDGE remains substantially above uniform sampling across all tested hyperparameter configurations on LVB and MLVU.Uniform sampling achieves 58.9% on LVB and 57.3% on MLVU.
  • Core hyperparameters: k_s varies by 1.4 percentage points on LVB and 1.1 percentage points on MLVU, the largest variation among the four core hyperparameters.k_s controls how much temporal context surrounds each detected event core.
  • Core hyperparameters: k_b varies by 0.7 percentage points on LVB and 1.1 percentage points on MLVU, indicating a smaller but useful boundary-frame contribution.Boundary frames usually occupy only a small fraction of the sampling budget.
  • Core hyperparameters: The default configuration provides the best average performance while remaining stable under reasonable perturbations of p_min and w_0.Very small p_min values can add spurious event cores, while overly large values can suppress weaker informative events; w_0 controls local-region coverage.
  • Safety clamps: Varying w_max from 6 to 15 and k_e from 2.0 to 6.0 changes accuracy by less than 0.5 percentage points on both benchmarks.These safety-clamp parameters are rarely activated in practice.

G Multi-Peak Analysis

RIDGE’s multi-peak behavior is associated with larger gains when multiple relevance peaks occur, while held-out configuration selection preserves its benchmark-wide advantage. Its selections are distributed beyond the strongest peak rather than concentrated in one window.

  • Multi-peak subsets: RIDGE improves over AKS by 2.80 percentage points on multi-peak instances, versus 0.39 percentage points on single-peak instances.The comparison uses LongVideoBench with Qwen2.5-VL-7B and N=32.
  • Selection distribution: Only 7.15% of RIDGE-selected frames fall inside the strongest-peak window on the non-short multi-peak subset.RIDGE does not guarantee that every detected peak receives a selected frame.
  • Held-out control: A configuration selected using only the 20% LVB development split achieves a four-benchmark average of 62.3 versus 62.2 for the paper default.It consistently outperforms uniform sampling across all four evaluation sets without access to reported evaluation labels.

I More Visualization Results

Qualitative examples show RIDGE correcting diverse long-video question-answering errors across Qwen2.5-VL-7B and LLaVA-OV-7B. The comparisons emphasize that answers may require temporal processes rather than isolated high-relevance frames.

  • Figure 6 examples: RIDGE corrects all six illustrated errors across action recognition, text reading, object identification, color discrimination, movie reference, and counting.Examples include correcting “No smoking” misread as “Safety first” and tires mistaken for boxes.
  • Figure 6 examples: The examples span Qwen2.5-VL-7B and LLaVA-OV-7B, with RIDGE shifting selection toward query-relevant moments.The base models otherwise select frames that miss decisive visual evidence and produce plausible but incorrect answers.
  • Temporal evidence: Answering the illustrated questions often requires a short temporal process containing buildup, the event core, and what follows it.This supports comparing methods by whether their selections cover the relevant temporal sequence rather than only one frame.
  • Figure 7 comparison: Figure 7 compares uniform sampling, AKS, FOCUS, and RIDGE using selected key frames or temporal spans and query-frame relevance trajectories.The qualitative comparison focuses on how each method selects evidence for the same questions.
Loading 2608.29958v1…