PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of MLLM-Agnostic Plug-and-Play Keyframe Selection Methods for Long Video Understanding

Dilip Sarkar, Md. Safayet Islam, Liang Liang

arXiv:2609.13250v1cs.CVcs.AI

TL;DR

Long videos can overwhelm MLLMs’ visual-token budgets, and earlier PaP methods lacked a common head-to-head comparison across models and benchmarks. This paper evaluates five methods across three MLLMs and three benchmarks; QAaF leads in 13 of 15 aggregate settings, with FOCUS second overall.

  • Problem

    MLLM token limits make long-video understanding difficult, while previously reported PaP methods had not been compared head-to-head across their differing benchmarks and MLLMs.

    Source: §1 · pp. 1–3

  • Method

    The paper compares five existing PaP keyframe-selection methods across three benchmarks and three MLLMs.

    Source: §1 · p. 3

  • Results

    QAaF achieves the best performance in 13 of 15 aggregate evaluation settings and second-best performance in the remaining two, while FOCUS ranks second overall.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The evaluation fills the head-to-head comparison gap and provides a common experimental reference for comparing training-free keyframe-selection methods.

    Source: §5 · p. 10

    PaP methods implicitly assume a frame-level cooperative property among video frames.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Multimodal large language models (MLLMs) cannot process every frame of a long video because of limitations in visual-token and computational budgets. Three primary approaches have been proposed to enhance their long-video understanding capabilities: (i) Retraining an MLLM on a large video corpus and/or extending its input length; (ii) Training an adapter for a specific MLLM that takes the entire video and the query as input and selects the most relevant video frames; and (iii) Developing a training-free, plug-and-play (PaP) adapter that is MLLM-agnostic. We refer to the third approach as PaP keyframe selection. A PaP method may use only candidate video frames without considering the query, or it may use both candidate video frames and the query. The first approach is prohibitively expensive. The second approach requires substantial training time and computational resources, but it is accessible to many because an adapter contains significantly fewer trainable parameters than an entire MLLM. The third approach has the lowest computational cost and is therefore broadly accessible. To the best of our knowledge, only five PaP methods have been reported within the past year. All of these methods have been evaluated on one or more video question-answering benchmarks and have demonstrated improvements in long-video understanding. However, the methods were evaluated on different benchmarks using different MLLMs. We present a comprehensive evaluation of these five methods using three MLLMs across three long-video understanding benchmarks. Our results show that QAaF achieves the best performance in 13 of the 15 aggregate evaluation settings, while FOCUS ranks second overall. These results provide a common experimental reference for comparing training-free keyframe-selection methods for MLLMs.

1 Introduction

MLLMs face token-budget limits that make long-video processing difficult, while uniformly spaced frames can miss query-critical cues. The paper addresses the lack of head-to-head comparisons by evaluating existing plug-and-play keyframe-selection methods across multiple MLLMs and benchmarks.

1 Introduction

  • A 5-minute, 24-fps video contains 7,200 frames, far beyond an MLLM’s limited token budget, and uniform sampling can omit query-critical visual cues.The 16 uniformly sampled frames may be diverse without including the frames most relevant to the query.Source: §1 · pp. 1–2
  • The three approaches are MLLM retraining or input-length extension, model-specific trained adapters, and training-free, MLLM-agnostic plug-and-play keyframe selection.PaP methods may select frames without the query or, more generally, use both candidate frames and the query.Source: §1 · p. 2
  • Retraining is prohibitively expensive, whereas trained adapters are less costly because they have fewer trainable parameters than an MLLM; PaP methods have the lowest computational cost.The paper describes PaP as broadly accessible, including to academic researchers.Source: §1 · p. 2
  • Existing PaP methods had been assessed on different benchmarks with different MLLMs and ablation protocols, leaving no head-to-head comparisons.The introduction names five methods: Q-Frame, AKS, FOCUS, MaxInfo, and QAaF.Source: §1 · pp. 2–3
  • The paper presents a comprehensive comparison of five PaP methods across three benchmarks and three MLLMs.It compares the methods’ strengths and weaknesses.Source: §1 · p. 3

2 Preliminaries and Problem Statement

The section defines the video, query, frame, and model notation before framing keyframe selection as choosing a limited set of frames to maximize the chance of a correct answer. It then describes why exact selection is difficult, how methods approximate frame value, and why candidate frames are sampled before query-aware selection.

2.1 Preliminaries

  • Keyframes are selected from a video-frame sequence, with the chosen subsequence and its keyframes denoted separately; a pretrained vision-language model maps frames and queries to same-dimensional token vectors.The MLLM takes the query or query-answer text together with selected keyframes and produces an answer.Source: §2.1 · p. 3

2.2 Problem Statement

  • The objective is to choose k, possibly non-consecutive, frames that maximize the MLLM’s probability of answering the query correctly, but exact selection faces an intractable combinatorial search and unavailable confidence estimates.The ideal selector is defined in terms of the MLLM’s estimated answer confidence.Source: §2.2 · p. 4
  • PaP methods approximate frame value with a single-frame contribution estimate and assume a cooperative property among frames to support individual frame evaluation.The section notes that this assumption extends from three frames to any number of frames.Source: §2.2 · p. 4
  • After scoring and sorting candidate frames, the polynomial-time selection procedure takes the first k frames as keyframes.The stated sorting time is O(Ts log Ts).Source: §2.2 · p. 4

2.3 Video Frame Sampling

  • Because decoding and evaluating every video frame is costly, practical selection first uses fixed-rate sampling to reduce the candidate pool, then selects query-relevant keyframes.Figure 3 depicts 16 frames with the highest query-aligned scores selected using QAaF.Source: §2.3 · p. 5, Figure 3 · p. 5

3 MLLM-Agnostic Training-Free PaP Methods

The section presents PaP keyframe selection as a modular, training-free approach, with methods that select frames using query relevance or, in MaxInfo’s case, visual diversity. It describes their selection strategies and evaluation setups.

3 MLLM-Agnostic Training-Free PaP Methods

  • PaP methods pair candidate-frame subsampling with selection from the reduced pool; most use query-aware alignment, while MaxInfo selects mutually dissimilar frames without the query.Source: §3 · p. 5

3.1 Known Keyframe Selection Methods

  • Q-Frame scores frame-query inner products and uses the Gumbel trick to select frames; its evaluations vary MLLMs, benchmarks, resolutions, and sampling settings.Source: §3.1 · p. 6
  • FOCUS treats video segments as bandit arms, estimates their relevance from sampled frames, and further evaluates promising segments before selecting keyframes.Its evaluation varies VLMs, MLLMs, benchmarks, and clip lengths; ablations examine two-stage exploration and the Bernstein confidence radius.Source: §3.1 · p. 6
  • AKS adds a temporal-range term to frame relevance, which may help when answering a query requires information from different video segments.Its evaluations use three VLMs with GPT-4, Qwen2-VL, and LLaVA-Video on LongVideoBench and Video-MME.Source: §3.1 · pp. 6–7
  • MaxInfo samples candidate frames, compresses their VLM feature vectors with truncated SVD, then uses MaxVol to select visually varied frames independently of the query.It is evaluated on Video-MME with three MLLMs, using CLIP, DINOv2, and SigLIP frame vectors.Source: §3.1 · p. 7
  • QAaF selects frames using cosine alignment with either the query or query-answer texts; its multiple-choice variant scores each answer-extended query against every frame.Both variants were evaluated with CLIP and LongCLIP across three MLLMs and three benchmarks.Source: §3.1 · p. 7

4 Experimental Evaluation

The evaluation applies five PaP keyframe-selection methods to three benchmarks and three MLLMs under a shared comparison protocol. QAaF leads in 13 of 15 aggregate settings, while benchmark-specific results reveal task- and duration-dependent trade-offs.

4.1 Experimental Setup

  • Three benchmarks—MLVU, Video-MME, and LongVideoBench—and three MLLMs were evaluated using a shared comparison protocol.The protocol fixed the frame budget at 16 for uniform sampling, AKS, FOCUS, MaxInfo, and QAaF, while retaining Q-Frame’s multi-resolution setting.Source: §4.1 · p. 7

4.2 Main Results

  • 13 of 15 aggregate settings: QAaF ranks best, while FOCUS ranks second overall across the evaluated methods.FOCUS has one best and seven second-best performances.Source: Table 2 · p. 8, §4.2 · p. 8

4.2.1 Results for MLVU benchmark dataset

  • 8.8, 7.8, and 5.11 points: QAaF beats uniform sampling on Qwen2-VL, LLaVA-Video, and InternVL3.5, respectively, but task-level winners vary.QAaF has the highest cross-model mean of 70.23, while uniform sampling remains strongest on TR and QaF leads NQA, AO, and AC.Source: Table 3 · p. 8, §4.2.1 · p. 9

4.2.2 Results for Video-MME benchmark dataset

  • 4.83/5.49, 2.25/2.63, and 1.48/2.11 points: QAaF beats uniform sampling without/with subtitles on Qwen2-VL, LLaVA-Video, and InternVL3.5.QAaF is strongest on short and medium videos, while FOCUS leads on long videos; subtitles improve most configurations, with some exceptions.Source: Table 4 · p. 9, §4.2.2 · pp. 9–10

4.2.3 Results for LongVideoBench benchmark dataset

  • 63.20/66.57: InternVL3.5 with QAaF + LongCLIP leads LongVideoBench without/with subtitles, and QAaF performs best in the longer duration splits.QAaF also improves over uniform sampling for Qwen2-VL and LLaVA-Video; subtitles raise its overall performance for all three MLLMs.Source: Table 5 · p. 10, §4.2.3 · p. 10

5 Conclusion

The paper evaluates five training-free, MLLM-agnostic frame-selection methods across three MLLMs and three benchmarks. QAaF leads in most aggregate settings, with FOCUS second overall.

  • QAaF achieves the best performance in 13 of 15 aggregate evaluation settings and second-best in the remaining two; FOCUS ranks second overall.Source: §5 · p. 10
  • The evaluation compares five training-free, MLLM-agnostic methods using three MLLMs across three benchmarks.Source: §5 · p. 10
  • Future work should include consistent ablation studies across all five methods.Source: §5 · p. 10
Loading 2609.13250v1…