PaperlayerView exact revision on arXiv

Source-linked AI summary

FOCUS: Efficient Keyframe Selection for Long Video Understanding

Zirui Zhu, Hailun Xu, Yang Luo, Yong Liu, Kanchan Sarkar, Zhenheng Yang, Yang You

arXiv:2510.27280v2cs.CVcs.AIcs.LG

TL;DR

Long videos overwhelm MLLMs with visual tokens, while downsampling and pre-filtering can miss informative frames. FOCUS selects query-relevant frames by exploring temporal clips as bandit arms under a strict budget. It improves accuracy across two benchmarks and four MLLMs while processing fewer than 2% of video frames.

  • Problem

    Long videos create overwhelming visual-token costs for MLLMs, while downsampling and pre-filtering can miss critical or informative frames.

    Source: §1 · pp. 1–2

  • Method

    FOCUS uses a training-free, model-agnostic bandit strategy that treats temporal clips as arms and applies confidence-guided exploration to select query-relevant frames under a strict budget.

    Source: §1 · p. 2, §4 · p. 10

  • Results

    FOCUS consistently improves accuracy across four MLLMs on two long-video QA benchmarks while processing fewer than 2% of video frames.

    Source: §1 · p. 3, §4 · p. 10

  • Takeaways & Limitations

    Lightweight, training-free keyframe selection guided by statistical principles can enhance the scalability and practicality of MLLMs for long-video understanding.

    Source: §4 · p. 10

    FOCUS assumes frame-level relevance scores are i.i.d., neglecting temporal dependencies between video segments.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Multimodal large language models (MLLMs) represent images and video frames as visual tokens. Scaling from single images to hour-long videos, however, inflates the token budget far beyond practical limits. Popular pipelines therefore either uniformly subsample or apply keyframe selection with retrieval-style scoring using smaller vision-language models. However, these keyframe selection methods still rely on pre-filtering before selection to reduce the inference cost and can miss the most informative moments. We propose FOCUS, Frame-Optimistic Confidence Upper-bound Selection, a training-free, model-agnostic keyframe selection module that selects query-relevant frames under a strict token budget. FOCUS formulates keyframe selection as a combinatorial pure-exploration (CPE) problem in multi-armed bandits: it treats short temporal clips as arms, and uses empirical means and Bernstein confidence radius to identify informative regions while preserving exploration of uncertain areas. The resulting two-stage exploration-exploitation procedure reduces from a sequential policy with theoretical guarantees, first identifying high-value temporal regions, then selecting top-scoring frames within each region. On two long-video question-answering benchmarks, FOCUS delivers substantial accuracy improvements while processing less than 2% of video frames. For videos longer than 20 minutes, it achieves an 11.9% gain in accuracy on LongVideoBench, demonstrating its effectiveness as a keyframe selection method and providing a simple and general solution for scalable long-video understanding with MLLMs. Code is available at https://github.com/NUS-HPC-AI-Lab/FOCUS.

1 INTRODUCTION

Long-video MLLMs face a token-cost challenge, while current selection pipelines can pre-filter away informative moments. FOCUS uses a bandit-based clip selection strategy to find query-relevant keyframes under a budget, improving benchmark accuracy.

1 INTRODUCTION

  • Long videos generate so many visual tokens that MLLM inference becomes computationally prohibitive.Uniform downsampling can miss critical content, while increasing frame rate causes token explosion.Source: §1 · p. 1
  • Existing relevance-scoring methods pre-filter frames to reduce cost, potentially excluding the most informative keyframes.For a one-hour video, exhaustive scoring is costly, prompting lower-frame-rate pre-filtering.Source: §1 · p. 2
  • FOCUS treats temporal clips as bandit arms and uses empirical relevance means and Bernstein confidence radii to select promising regions.It allocates computation to clips that are promising or uncertain, then selects top-relevance frames within chosen clips.Source: §1 · p. 2
  • 11.9% accuracy improvement on LongVideoBench videos longer than 20 minutes, while FOCUS also cuts inference cost.Across two benchmarks, it improves accuracy over state-of-the-art keyframe-selection baselines.Source: §1 · p. 2
  • FOCUS is training-free and model-agnostic, and its gains are consistent across two benchmarks and four popular MLLMs.The method is presented as a plug-and-play keyframe-selection module.Source: §1 · pp. 2–3

2 METHOD

FOCUS approximates query-relevant keyframe selection as a budgeted clip-level bandit problem, using encoder relevance scores to guide exploration. It replaces sequential arm selection with a batched two-stage procedure, then selects frames within the chosen clips.

2.1 PROBLEM FORMULATION

  • Because evaluating all frame utilities is intractable, FOCUS uses bounded, unbiased encoder relevance scores as proxies and allocates a limited sampling budget across clips.The frame-level objective is approximated through a combinatorial pure-exploration formulation at the clip level.Source: §2.1 · p. 3

2.2 CLIP-LEVEL SELECTION AS MULTI-ARMED BANDIT

  • Each fixed-length, non-overlapping clip is a bandit arm; the method seeks promising arms before selecting keyframes, assuming equal keyframe allocation across selected arms.The paper states that this size-m-subset and equal-allocation setting yields a regret bound and is effective in experiments.Source: §2.2 · p. 4

2.3.1 OPTIMAL ARM SELECTION.

  • The sequential selector compares lower confidence bounds for current top-m clips with upper bounds for others, sampling the most uncertain contender until the top-m set stabilizes.Although it returns the optimal top-m set with high probability, sequential arm pulls cannot be parallelized and waste GPU utilization.Source: §2.3.1 · p. 5

2.3.2 TWO-STAGE ARM SELECTION.

  • The batched variant explores all arms with q pulls, then gives z additional pulls to the top αm optimistic arms and selects the final m using empirical means.The two-stage design is introduced to make the procedure practical and easier to parallelize.Source: §2.3.2 · p. 6

2.4 FRAME SELECTION WITHIN SELECTED ARMS

  • Within selected arms, FOCUS allocates the frame budget equally, interpolates observed rewards by nearest neighbors, then samples frames without replacement using reward-based distributions.Per-arm allocations are adjusted to sum to the total frame budget.Source: §2.4 · p. 6

3 EXPERIMENTS

Across two long-video QA benchmarks, FOCUS improved accuracy over uniform sampling and state-of-the-art keyframe baselines, with stronger gains on longer videos. It also reduced frame-scoring cost, while accuracy-efficiency varied with α.

3.1 EXPERIMENTAL SETUP

  • Evaluation uses zero-shot multiple-choice QA on LongVideoBench and Video-MME, with subtitles disabled, frozen model parameters, and frame-selection strategy varied.The experiments include four open-source MLLMs and GPT-4o; BLIP ITM scores frame relevance.Source: §3.1 · p. 6

3.2 PERFORMANCE ANALYSIS

  • FOCUS consistently outperforms uniform sampling across four MLLMs on both benchmarks, with LongVideoBench gains of 3.2%, 6.7%, 5.9%, and 4.6%.The corresponding Video-MME gains are 0.7%, 2.1%, 1.8%, and 1.0%; Figure 3 marks query-informative frames, many captured by FOCUS.Source: Table 1 · p. 7, §3.2 · p. 7, Figure 3 · p. 8
  • Qwen2-VL-7B with FOCUS outperforms Gemini-1.5-Flash on LongVideoBench while using 8× fewer input frames.The paper reports that larger MLLMs with more frame inputs tend to perform better, while FOCUS narrows this gap.Source: §3.2 · p. 7
  • FOCUS assumes frame-level relevance scores are i.i.d., neglecting temporal dependencies between video segments; accounting for these shifts the problem to other bandit settings.The paper leaves Lipschitz/metric and contextual bandit extensions for future work.Source: §3.2 · p. 7

3.3 COMPARISON WITH STATE-OF-THE-ART

  • 11.9% over uniform sampling and 7.6% over Top-K on LongVideoBench videos longer than 20 minutes; Video-MME gains are 1.8% and 1.4%, respectively.Table 2 compares methods at matched keyframe counts across video-length buckets.Source: Table 2 · p. 8, §3.3 · p. 9
  • 1.6% of BLIP forward passes and 5.5 GPU hours make FOCUS the most efficient method while achieving the best overall performance.AKS without pre-filtering is estimated to require over 255 GPU hours; pre-filtering reduces that cost to 9.3 GPU hours.Source: Table 3 · p. 9, §3.3 · p. 9
  • On short videos, keyframe methods perform similarly and outperform uniform sampling; the authors attribute this possibly to LLaVA-Video-7B reasoning saturation.The proposed explanation is that input selection plays a less critical role when the underlying model's reasoning capabilities are saturated.Source: §3.3 · p. 9

3.4 EFFICIENCY-ACCURACY TRADE-OFF

  • At α = 0.25, FOCUS evaluates 1.6% of frames in 5.5 GPU hours at 63.5% accuracy; α = 0.5 reaches 63.6% at 2.5% of frames and 9.2 GPU hours.The authors describe the accuracy increase as negligible relative to the higher cost, indicating diminishing returns from exploring more arms.Source: Table 4 · p. 9, §3.4 · p. 9

4 CONCLUSION

FOCUS addresses long-video token growth with a coarse-to-fine, bandit-based keyframe selector. Experiments report higher accuracy across four MLLMs on two long-video QA benchmarks while processing fewer than 2% of video frames.

4 CONCLUSION

  • The method treats temporal clips as bandit arms, uses empirical means and Bernstein confidence bounds to identify relevant regions, and selects frames in a coarse-to-fine two-stage procedure.The procedure preserves optimism while enabling parallel inference.Source: §4 · p. 10
  • FOCUS improves accuracy across four MLLMs on two long-video QA benchmarks while processing fewer than 2% of video frames.The conclusion describes this as enhancing the scalability and practicality of MLLMs for long-video understanding.Source: §4 · p. 10

A.1.1 MULTIMODAL LARGE LANGUAGE MODELS (MLLMS) FOR VIDEO UNDERSTANDING

  • Long-video MLLM work includes token compression, streaming, selective analysis, hierarchical control, and long-context training alongside model and architecture advances.The cited examples include sparse memory, condensed tokens, incremental processing, and extended-context approaches.Source: §A.1.1 · pp. 16–17

A.1.2 VISION-LANGUAGE PRETRAINED MODELS

  • Vision-language pretraining spans fusion architectures, dual encoders, and hybrid objectives; video extensions use hierarchical encoding, sparse sampling, and large image-text models.Examples include CLIP-based video matching and frozen-language-model approaches for zero-shot VQA.Source: §A.1.2 · p. 17

A.1.3 KEYFRAME SELECTION

  • Keyframe selection includes training-free query-relevance methods, instruction-aligned learned selectors, and task-agnostic summarization approaches that may miss query-critical frames.Training-free examples use similarity, temporal coverage, or hierarchical search; learned methods use LLM/MLLM feedback.Source: §A.1.3 · pp. 17–18

A.1.4 MULTI-ARMED BANDITS AND BATCHED EXPLORATION

  • Bandit methods distinguish regret minimization from pure exploration, with combinatorial pure exploration identifying high-value subsets and batched methods reducing adaptivity rounds.The cited literature describes confidence-bound and elimination algorithms, plus batch procedures for resource-constrained exploration.Source: §A.1.4 · p. 18

B BERNSTEIN CONFIDENCE RADIUS

This section defines an empirical Bernstein confidence radius for arm estimates and states a probabilistic bound under bounded rewards.

B BERNSTEIN CONFIDENCE RADIUS

  • The theorem states a high-probability bound for the arm estimates under the frame-query relevance setting.The proof invokes Bernstein’s inequality because rewards are bounded in [0, 1].Source: B BERNSTEIN CONFIDENCE RADIUS · pp. 18–19
  • The empirical Bernstein confidence radius uses an arm’s empirical variance and pull count to quantify uncertainty around its empirical mean.The radius is defined for arm a at round n.Source: B BERNSTEIN CONFIDENCE RADIUS · p. 18

C REGRET BOUND

The regret analysis connects successful identification of the oracle top-s arm set to small frame-level regret, while leaving a tighter bound for future work.

C REGRET BOUND

  • Algorithm 2 returns the oracle top-s set with a stated probability guarantee based on confidence intervals for the arms.At termination, the proof uses the condition that each arm’s mean lies within its confidence radius of its empirical estimate.Source: C REGRET BOUND · p. 19
  • Obtaining the oracle top-s set guarantees small frame-level regret, defined as the gap between optimal and selected frame rewards.Because frame-level sampling is finite, the top-k frames with the highest rewards can be found.Source: C REGRET BOUND · p. 19
  • A tighter frame-level regret bound is left for future work.Source: C REGRET BOUND · p. 19

D VISUALIZATIONS OF FAILURE CASES

The failure-case analysis distinguishes MLLM reasoning failures despite correct frame selection from cases where FOCUS misses a brief, critical transition.

D VISUALIZATIONS OF FAILURE CASES

  • The two examples represent distinct failure patterns: MLLM difficulty reasoning over similar charts and adaptive sampling missing crucial information in intrinsically challenging cases.Source: D VISUALIZATIONS OF FAILURE CASES · pp. 19–20
  • In one case, FOCUS selects the informative frames, but the MLLM fails to distinguish similar chart content because of limited reasoning and perception.The chart appears multiple times among other background content.Source: D VISUALIZATIONS OF FAILURE CASES · pp. 19–20
  • In another case, FOCUS misses critical frames in a 10-minute video because the relevant segment lasts only 1–2 seconds.The passage says such a compact transition can make the keyframes difficult to identify even for human experts.Source: D VISUALIZATIONS OF FAILURE CASES · p. 20, Figure 4 · p. 20

E COMPARISON WITH STATE-OF-THE-ART

FOCUS outperforms AKS and Q-Frame across three backbones on LongVideoBench and Video-MME under the same frame budget. The comparison also highlights differences in how the baselines select frames.

E COMPARISON WITH STATE-OF-THE-ART

  • FOCUS improves Qwen2-VL-7B, LLaVA-OV-7B, and LLaVA-Video-7B by 4.6–6.7% on LongVideoBench and up to 2.3% on Video-MME.It outperforms AKS and Q-Frame across all three backbones on both benchmarks under the same frame budget.Source: Table 5 · p. 20, E COMPARISON WITH STATE-OF-THE-ART · p. 21
  • AKS downsamples to one frame per second before scoring and allocates frame slots across segments, while Q-Frame uses similarity-based stochastic selection and heterogeneous resolutions.Both are described as training-free keyframe-selection approaches.Source: E COMPARISON WITH STATE-OF-THE-ART · p. 21
  • AKS consistently outperforms Q-Frame on both benchmarks where Q-Frame is evaluated; the authors attribute this to AKS balancing query relevance with temporal coverage.They contrast this with Q-Frame's lack of explicit temporal sampling or search design for locating informative moments.Source: E COMPARISON WITH STATE-OF-THE-ART · p. 21

F EXPERIMENTS ON MORE BENCHMARKS

FOCUS improves results on MLVU and VSI-Bench, extending evaluation beyond long-video QA benchmarks. Its gains are smaller on VSI-Bench, where videos are shorter and informative content is more uniformly distributed.

F EXPERIMENTS ON MORE BENCHMARKS

  • On MLVU, Qwen2-VL-7B rises from 59.7% to 67.0% (+7.3%) and LLaVA-Video-7B from 68.2% to 72.7% (+4.5%) with FOCUS.These gains are +2.7% and +1.5% over AKS, respectively.Source: Table 6 · p. 21, F EXPERIMENTS ON MORE BENCHMARKS · p. 22
  • On VSI-Bench, FOCUS raises Qwen2-VL-7B accuracy from 36.5% to 39.0% (+2.5%) and LLaVA-Video-7B from 41.7% to 42.4% (+0.7%).The benchmark emphasizes fine-grained spatial reasoning over relatively short egocentric clips.Source: Table 6 · p. 21, F EXPERIMENTS ON MORE BENCHMARKS · p. 22
  • MLVU covers nine tasks across 1,730 long videos, while VSI-Bench evaluates spatial understanding and memory on 288 egocentric indoor videos.MLVU videos range from 3 minutes to 2 hours; VSI-Bench includes over 5,000 question–answer pairs.Source: F EXPERIMENTS ON MORE BENCHMARKS · pp. 21–22
  • The authors identify smaller improvements on VSI-Bench as a limitation when videos are short and informative content is more uniformly distributed.They suggest spatially-aware frame selection in low-redundancy settings as a future direction.Source: F EXPERIMENTS ON MORE BENCHMARKS · p. 22

G.1 TWO-STAGE EXPLORATION-EXPLOITATION

The ablations find that coarse region selection and fine-grained exploration contribute complementary benefits, while Bernstein confidence bounds outperform empirical-mean selection. Together, these tests examine the two-stage procedure and its uncertainty-aware exploration.

G.1 TWO-STAGE EXPLORATION-EXPLOITATION

  • The full two-stage FOCUS achieves the best performance across all three backbones, gaining up to 2.3% over the single-stage variants.FOCUS-C uses coarse arm selection followed by random within-arm sampling; FOCUS-F explores uniformly across the video without arm-level preselection.Source: Table 7 · p. 22, §G.1 · p. 22

G.2 BERNSTEIN CONFIDENCE RADIUS

  • The ablation compares ranking arms and selecting frames by empirical mean alone against using a variance-aware Bernstein confidence radius.The authors describe Bernstein's radius as more robust to high-variance rewards than classical UCB.Source: §G.2 · p. 23, Table 8 · p. 23
  • FOCUS-M improves over uniform sampling, but the full method with Bernstein confidence radius performs best, improving by up to 2.6% over uniform and base models.The authors attribute the advantage to additional exploration of high-uncertainty clips, especially those with diverse or rapidly changing scenes.Source: Table 8 · p. 23, §G.2 · p. 23

G.3 EFFECT OF CLIP LENGTH

FOCUS remains effective across clip-length settings and vision-language encoders, with trade-offs between accuracy and selection cost. The reported encoder comparison finds BLIP strongest, while CLIP and SigLIP also improve over uniform sampling.

G.3 EFFECT OF CLIP LENGTH

  • All tested clip lengths outperform uniform sampling (58.9% vs. 62.3–63.7%); 8-second clips give slightly better accuracy but cost more, while 32-second clips reduce GPU hours.The authors identify 16 seconds as a practical accuracy-efficiency trade-off.Source: Table 9 · p. 23, §G.3 · p. 23

G.4 EFFECT OF VISION-LANGUAGE ENCODER

  • FOCUS uses a frame-query relevance module that can be integrated with different vision-language encoders; the reported evaluation includes CLIP, SigLIP, and BLIP.The main experiments use BLIP to align with the AKS baseline.Source: §G.4 · pp. 23–24
  • All three encoders improve over uniform sampling; BLIP performs strongest, while CLIP and SigLIP provide 1.3% and 2.0% gains, respectively.The comparison uses LongVideoBench with LLaVA-Video-7B.Source: Table 10 · p. 24, §G.4 · p. 24

H LIMITATIONS

FOCUS does not model temporal dependence between frame-query relevance scores, and the authors report that LLMs were used only for proofreading and light copy-editing.

H LIMITATIONS

  • The method assumes frame-query relevance scores are i.i.d., without modeling temporal dependence between frames.Source: H LIMITATIONS · p. 24
  • Strong correlations across video parts may violate the i.i.d. assumption; the authors leave alternative bandit formulations for future work.They suggest Lipschitz/metric or contextual bandits for this setting.Source: H LIMITATIONS · p. 24

I THE USE OF LARGE LANGUAGE MODELS (LLMS)

Loading 2510.27280v2…