PaperlayerView exact revision on arXiv

Source-linked AI summary

FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understanding

De-An Huang, Subhashree Radhakrishnan, Zhiding Yu, Jan Kautz

arXiv:2504.17447v1cs.CVcs.AI

TL;DR

Training long-context LMMs poses challenges. FRAG uses an LMM to score each frame or page independently for query relevance, then feeds the Top-K items to an LMM for answer generation without fine-tuning. FRAG improves long-video and long-document understanding and achieves state-of-the-art results across both; InternVL2-76B improves by 5.8% on MLVU.

  • Problem

    Training long-context LMMs poses challenges.

    Source: §1 · p. 1

  • Method

    FRAG uses an LMM to score each frame or page independently for query relevance, then feeds the Top-K items to an LMM for answer generation without fine-tuning.

    Source: §1 · p. 2

  • Results

    FRAG improves long-video and long-document understanding and achieves state-of-the-art results across both; InternVL2-76B improves by 5.8% on MLVU.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Independent frame scoring can be parallelized across devices and avoids quadratic self-attention processing all frames jointly.

    Source: §1 · p. 2

    With 1024 sampled frames, performance regresses because unconstrained Top-K can concentrate selections in a short time period; frame diversity may improve this.

    Source: §4.3.2 · p. 8

Abstract

from arXiv · show

There has been impressive progress in Large Multimodal Models (LMMs). Recent works extend these models to long inputs, including multi-page documents and long videos. However, the model size and performance of these long context models are still limited due to the computational cost in both training and inference. In this work, we explore an orthogonal direction and process long inputs without long context LMMs. We propose Frame Selection Augmented Generation (FRAG), where the model first selects relevant frames within the input, and then only generates the final outputs based on the selected frames. The core of the selection process is done by scoring each frame independently, which does not require long context processing. The frames with the highest scores are then selected by a simple Top-K selection. We show that this frustratingly simple framework is applicable to both long videos and multi-page documents using existing LMMs without any fine-tuning. We consider two models, LLaVA-OneVision and InternVL2, in our experiments and show that FRAG consistently improves the performance and achieves state-of-the-art performances for both long video and long document understanding. For videos, FRAG substantially improves InternVL2-76B by 5.8% on MLVU and 3.7% on Video-MME. For documents, FRAG achieves over 20% improvements on MP-DocVQA compared with recent LMMs specialized in long document understanding. Code is available at: https://github.com/NVlabs/FRAG

1. Introduction

Long-context LMMs face costly data and computation, motivating the question of whether long inputs can instead be handled by independently identifying relevant frames or pages. FRAG uses zero-shot LMM scoring and Top-K selection, and the paper reports improvements across long-video and long-document benchmarks.

1. Introduction

  • Long-context LMMs require costly high-quality training data and computation, and are often limited to 7B models unlike typical LMMs that scale beyond 70B.The paper says these challenges limit the capabilities of existing long-context LMMs.Source: §1 · p. 1
  • Long inputs need not be processed globally when a query can be answered from relevant moments or pages identified independently.The introduction illustrates this with black-backpack video frames and pie-chart slide pages.Source: §1 · pp. 1–2
  • FRAG first scores each frame or page independently for query relevance, then uses the highest-scoring items as input for final answer generation.This avoids processing the whole long input with a long-context LMM.Source: §1 · p. 2
  • Existing LMMs can score frames or pages zero-shot by estimating whether each contains sufficient information to answer the query.FRAG uses the probability of a “yes” response for ranking, without further tuning.Source: §1 · p. 2
  • Independent scoring avoids long-context training data, accommodates different model sizes, reduces joint-processing computation, and can be parallelized across devices.The paper contrasts scoring frames independently with processing them all jointly, where quadratic self-attention applies.Source: §1 · p. 2
  • FRAG achieves state-of-the-art results across long-video and multi-page document benchmarks, including a 5.8% improvement for InternVL2-76B on MLVU.The introduction also reports a 3.7% improvement on Video-MME, outperforming GPT-4o on LongVideoBench, and over 20% improvement on MP-DocVQA versus recent specialized LMMs.Source: §1 · p. 2

2. Related Work

Prior work addresses long video and multi-page document understanding through trained compression or selection methods, longer context, and document-specific visual processing. FRAG contrasts with these approaches by using zero-shot selection without further tuning.

2. Related Work

  • Long-video methods either train memory or resampler modules with LMMs or extend context length, which can limit model size because of computational burden.FRAG instead uses selected frames without further tuning and does not rely on long-context modeling.Source: §2 · p. 2
  • Multi-page document specialists use OCR, detection, and fine-grained supervision, while LMM work often adapts resolution; FRAG reduces visual tokens through question-relevant frame selection.This reduction can support high-resolution visual processing of relevant areas.Source: §2 · p. 2
  • Recent LMM frame-selection approaches may depend on proprietary language models or trained, application-specific selectors, whereas FRAG is zero-shot and applicable across applications.The related work identifies SeViLA as training a selector using LMMs.Source: §2 · p. 2
  • FRAG’s scoring and answering stages may use the same existing LMM or different ones, with neither requiring tuning.The overview depicts scoring sampled frames, selecting Top-K, and passing those frames to an answering LMM.Source: Figure 2 · p. 3

3. Frame Selection Augmented Generation

FRAG independently scores sampled frames or pages for query relevance, selects the Top-K, and supplies them to an LMM for answer generation. The paper describes this simple selection as effective for both long videos and documents.

3.1. Independent Frame Scoring

  • FRAG treats uniformly downsampled individual frames as proposals, avoiding the redundant information that temporally adjacent multi-frame segments may include within a fixed selection budget.The paper reports that single-frame proposals work well with Top-K selection.Source: §3.1 · p. 3
  • A query-conditioned LMM scores each proposal zero-shot by answering whether the image contains sufficient information; the probability of “yes” serves as the relevance score.A frame’s score may be below 0.5 even when it indicates relevance, since one frame may not answer a whole-video question.Source: §3.1 · p. 3

3.2. Selection for Generation

  • FRAG selects the Top-K proposals, restores temporal or page order, and feeds them to an answering LMM; Top-K can tolerate imperfect scores because the best page may still be selected.Selected video frames are treated as images, allowing multi-image LMMs to be used without video-specific input formats.Source: §3.2 · pp. 3–4
  • FRAG substantially improves over uniform sampling on long-video results, especially on LongVideoBench and MLVU.Table 1 presents the long-video understanding comparison.Source: Table 1 · p. 4

4. Experiments

The experiments evaluate FRAG on both long-video and long-document understanding tasks.

4. Experiments

  • FRAG selects relevant frames before generating outputs, and the experiments evaluate both long-video and long-document understanding.Source: §4 · p. 4

4.1. Evaluating Long Video Understanding

FRAG improves long-video understanding across most evaluated settings, and its models outperform existing methods, including long-context and frame-selection approaches.

4.1.1. Datasets and Metrics

  • The evaluation covers five video datasets, including three long-video benchmarks, egocentric EgoSchema, and NeXT-QA, with question-answering accuracy reported for each.The first three datasets average over eight minutes, EgoSchema around two minutes, and NeXT-QA 0.8 minutes.Source: §4.1.1 · p. 4

4.1.2. Implementation Details

  • Experiments use LLaVA-OneVision and InternVL2 at multiple sizes, scoring 256 uniformly sampled frames before selecting Top-32, except Top-24 for InternVL2-76B.Video answering disables high-resolution tiling to allocate computation across distinct frames; Video-MME uses 28 frames for InternVL2-40B when subtitles are included.Source: §4.1.2 · p. 4

4.1.3. Effectiveness of Frame Selection

  • Compared with Uniform-32 (uniformly sampled 32 frames), FRAG-Top32-256 (top 32 of 256 uniformly sampled frames) substantially improves most results, especially on LongVideoBench and MLVU; it improves all evaluated LMMs on Video-MME and consistently improves EgoSchema.On NeXT-QA, FRAG performs similarly to Uniform-32; its videos are less than a minute long, so 32 frames may already capture most of the information.Source: §4.1.3 · pp. 4–5

4.1.4. Comparison with SOTA

  • FRAG models outperform existing methods across long-video benchmarks, including GPT-4o on LongVideoBench and MLVU; FRAG-LLaVA-OV performs better overall than FRAG-InternVL2.FRAG-InternVL2-40B substantially outperforms existing methods on all datasets, while FRAG-LLaVA-OV-7B outperforms frame-selection methods using proprietary models.Source: Table 2 · p. 5, §4.1.4 · p. 5

4.2. Evaluating Long Document Understanding

FRAG improves long-document understanding over uniform sampling and existing methods across three benchmarks, including specialist models on SlideVQA and MP-DocVQA.

4.2.1. Datasets and Metrics

  • The benchmarks span 20-page SlideVQA, denser MP-DocVQA documents averaging 8.3 pages, and MMLongBench-Doc averaging 47.5 pages with cross-page and unanswerable questions.Their metrics include exact match and F1 for SlideVQA, accuracy and ANLS for MP-DocVQA, and accuracy and F1 for MMLongBench-Doc.Source: §4.2.1 · pp. 5–6

4.2.3. Effectiveness of Frame Selection

  • Against Uniform-32, FRAG improves SlideVQA by over 20%, MP-DocVQA by over 40%, and more than doubles F1 on MMLongBench-Doc.For SlideVQA and MP-DocVQA, Uniform-32 uses all pages when documents contain fewer than 32 pages; the comparison matches maximum tile budgets.Source: Table 3 · p. 6, §4.2.3 · p. 6

4.2.2. Implementation Details

  • For documents, FRAG scores all pages and selects Top-2; answering retains high-resolution processing, with tile limits of 16 or 12 for InternVL2-76B.The same LMM families are used as for videos, and the evaluation follows official prompting without FRAG-specific prompts.Source: §4.2.2 · p. 6

4.2.4. Comparison with SOTA

  • FRAG's smallest models improve SlideVQA F1 by 16% over the best existing methods, while FRAG-InternVL2-76B reaches a 28% total improvement.The cited comparison is 75.6 versus 47.3; FRAG is not trained on SlideVQA, unlike the supervised specialist baselines.Source: §4.2.4 · p. 6, Table 4 · p. 7
  • FRAG's 7B/8B models improve MP-DocVQA ANLS by 10% over mPLUG-DocOwl-2, and the 76B model further improves ANLS by 9%.mPLUG-DocOwl-2 trains on MP-DocVQA, whereas FRAG is zero-shot and ranks first on the leaderboard described in the paper.Source: §4.2.4 · pp. 6–7, Table 5 · p. 7
  • On MMLongBench-Doc, FRAG-InternVL2-8B almost triples both accuracy and F1 versus the best LMM below 14B; FRAG's 40B/76B models outperform GPT-4V.The benchmark averages almost 50 pages per document, and FRAG still significantly outperforms the stronger InternVL-2 baseline.Source: Table 6 · p. 7, §4.2.4 · p. 7
  • FRAG significantly outperforms existing methods on all three document benchmarks, and InternVL2 generally performs better than LLaVA-OV for documents.The authors contrast this model-family pattern with videos, where LLaVA-OV performs better.Source: §4.2.4 · p. 7

4.3. Ablation Studies

The ablations show that stronger scoring LMMs select better frames, while performance depends on the number of sampled and selected frames. The reported settings use 256 sampled frames and select 32 for answering.

4.3.1. Evaluating Scoring LMM.

  • With InternVL2-40B answering, InternVL2-76B is the best scorer; SigLIP is marginally above uniform on average but hurts three of five benchmarks.InternVL2-1B slightly improves over SigLIP and hurts performance less on the shorter EgoSchema and NeXT-QA videos, where SigLIP drops over 3%.Source: §4.3.1 · p. 7, Table 7 · p. 8

4.3.2. Evaluating Number of Sampled Frames

  • Performance improves through 512 sampled frames and regresses at 1024; NeXT-QA peaks at 64 because its 0.8-minute videos make even 128 frames oversampling.The authors attribute regression at higher sampling counts to Top-K selection concentrating frames in short time periods.Source: §4.3.2 · pp. 7–8, Figure 4 · p. 8

4.3.3. Evaluating Number of Selected Frames

  • With 256 sampled frames fixed, performance peaks at 32 selected frames—the number of input frames to the answering LMM—and the authors use 32 in their main experiments.This trend is also reported in other works cited by the authors.Source: §4.3.3 · p. 8, Figure 5 · p. 8

5. Conclusion

FRAG processes long inputs without long-context LMMs by selecting relevant frames or pages. The authors report zero-shot state-of-the-art results for long-video and long-document understanding.

5. Conclusion

  • FRAG uses zero-shot Top-K selection to achieve state-of-the-art long-video and long-document understanding without long-context LMMs.The conclusion identifies better selection strategies and improved efficiency as future work.Source: §5 · p. 8

A. Selection Scoring Prompt

The scoring prompt asks whether an image contains enough information to answer the query. It presents yes and no as the response choices.

A. Selection Scoring Prompt

  • The scoring prompt asks whether an image contains the details needed to answer the query, with yes/no response choices.The prompt requests the option letter directly.Source: A. Selection Scoring Prompt · p. 11

B. Analysis of Number of Sampled Frames

The analysis links oversampling to concentrated frame selection and compares cases where 512 or 1024 sampled frames produce the correct answer. The diversity advantage is clearer in examples where 512 succeeds and 1024 fails than in the reverse cases.

B. Analysis of Number of Sampled Frames

Loading 2504.17447v1…