PaperlayerView exact revision on arXiv

Source-linked AI summary

Frame-Voyager: Learning to Query Frames for Video Large Language Models

Sicheng Yu, Chengkai Jin, Huanyu Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xiaolei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, Qianru Sun

arXiv:2410.03226v4cs.CVcs.CL

TL;DR

Video-LLMs face input-token limits, and uniform sampling or individual text-frame matching may miss query-relevant information and relationships across frames. FRAME-VOYAGER learns to select frame combinations using rankings derived from a pre-trained Video-LLM’s prediction losses. Experiments report improvements over conventional frame-selection methods and state-of-the-art results in the evaluated settings.

  • Problem

    Video-LLMs are constrained by input-token limits, while uniform sampling and individual text-frame matching can miss information density variation or complex temporal relationships.

    Source: §1 · p. 1

  • Method

    FRAME-VOYAGER learns to select frame combinations, trained with rankings based on a pre-trained Video-LLM’s prediction loss for each combination.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Experiments report that FRAME-VOYAGER significantly boosts baseline Video-LLMs, achieves state-of-the-art results, and outperforms other frame-selection methods.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    FRAME-VOYAGER is presented as a plug-and-play frame-selection module that improves Video-LLM performance across the evaluated settings.

    Source: §1 · p. 2, §5 · p. 10

    Resource constraints meant ranking data were generated only with VILA-8B, precluding experiments with more powerful Video-LLMs that might yield superior results.

    Source: LIMITATIONS · p. 10

Abstract

from arXiv · show

Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos or the complex instructions in the tasks, leading to sub-optimal performance. In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task. To train Frame-Voyager, we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM. Given a video of M frames, we traverse its T-frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM's prediction losses. Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses. In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs. The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings, highlighting its potential as a plug-and-play solution for Video-LLMs.

1 INTRODUCTION

Video-LLMs cannot process entire videos within their token limits, while common frame-selection methods can miss query-relevant information or relationships across frames. FRAME-VOYAGER learns to select frame combinations, using Video-LLM prediction losses as ranking supervision, and improves performance over conventional selection methods in evaluated settings.

1 INTRODUCTION

  • Video-LLMs cannot process entire videos because of LLM token-length limits, and extending context can add computational complexity and the “lost-in-the-middle” issue.The paper identifies the input-length limit as a key challenge for video understanding.Source: §1 · p. 1
  • Uniform sampling misses variation in video information density, while text-frame matching can miss frame interactions and temporal relationships needed for complex tasks.Different questions may rely on distinct segments; some tasks require tracking action progression or cause-and-effect relationships.Source: §1 · pp. 1–2
  • FRAME-VOYAGER learns to query subsets of frames as combinations rather than retrieving individual frames separately.The paper motivates this approach as a way to account for dynamic scenes and global event context.Source: §1 · p. 2
  • The training pipeline ranks frame combinations by a pre-trained Video-LLM’s prediction loss, then uses the rankings to train FRAME-VOYAGER.For M video frames and an input limit of T frames, the pipeline evaluates C(M, T) combinations; lower-loss combinations rank higher.Source: §1 · p. 2
  • FRAME-VOYAGER significantly improves performance over uniform sampling and text-frame retrieval, especially on tasks requiring reasoning and information synopsis in long videos.The method is presented as a plug-and-play module for different Video-LLM architectures.Source: §1 · p. 2

2 RELATED WORK

Prior frame-selection methods either overlook query information or focus on individual-frame relevance without accounting for relationships among frames. FRAME-VOYAGER instead treats a frame combination as a whole when seeking frames that answer a query within input-length limits.

2 RELATED WORK

  • Transition-frame selection and clustering seek central frames but can overlook information from the query.These approaches are discussed alongside methods that match frames to query semantics.Source: §2 · p. 3
  • Text-frame matching selects frames semantically close to a query but ignores frame-to-frame relationships needed for complex temporal understanding.The paper also notes that grounded video question-answering methods target specific continuous segments, which may not suit general video question answering.Source: §2 · p. 3
  • FRAME-VOYAGER treats frames as a combination and seeks the combination that best answers the query under frame-length limits.The paper presents this as the first approach, to its knowledge, to consider frame combinations as a whole.Source: §2 · p. 3

3 FRAME-VOYAGER

FRAME-VOYAGER treats query-conditioned frame selection as a ranking problem, using reference Video-LLM losses to supervise a plug-and-play module that scores frame combinations and selects frames.

3 FRAME-VOYAGER

  • FRAME-VOYAGER formulates frame selection as ranking T-frame subsets for a video-query pair, rather than directly searching for an optimal combination.The training dataset is generated by a pre-trained reference Video-LLM.Source: §3 · p. 3

3.1 DATA COLLECTION

  • A human-free pipeline scores T-frame combinations with a reference Video-LLM’s ground-truth-answer language-modeling loss and ranks them from lowest loss upward.Exhaustive enumeration grows rapidly; experiments found training on smaller combinations such as C(16, 2) or C(32, 4) generalizes to larger inference settings.Source: §3.1 · pp. 3–4

3.2 MODEL TRAINING AND INFERENCE

  • The module processes all candidate frames with the query through bottom LLM layers to model query-frame and frame-to-frame interactions, then computes frame rewards for combinations.Frame and query features are formed from visual-token averages and query embeddings; combination reward averages the rewards of its constituent frames.Source: §3.2 · pp. 4–5
  • Training samples ranked combinations into pairs with the lower-loss combination preferred; inference selects the top T rewards in original temporal order without updating parameters.The inference procedure uses the same process as training through frame-reward computation.Source: §3.2 · pp. 5–6, Figure 2 · p. 5

4 EXPERIMENTS

Experiments evaluate FRAME-VOYAGER across Video Question Answering benchmarks and frame-selection settings. It improves on strong baselines, while analyses identify how frame count, component reuse, and question type affect performance.

4.1 EXPERIMENT SETTINGS

  • Evaluation covers four benchmarks, with 8-frame combinations selected from 128 or 16 candidates; training data comes from NextQA and VideoChatGPT.The study evaluates VILA-8B and VILA-40B, and reports Video-MME accuracy both with and without subtitles.Source: §4.1 · pp. 6–7

4.2 RESULTS AND ANALYSIS

  • 3.0% over vanilla VILA-8B on Video-MME without subtitles, with a 3.6% gain on long videos; 8-frame FRAME-VOYAGER also surpasses LongVILA using 128 or 256 frames.FRAME-VOYAGER also consistently improves larger Video-LLMs and is compatible with different architectures.Source: §4.2 · pp. 7–8
  • FRAME-VOYAGER outperforms all tested frame-extraction baselines on Video-MME, including uniform sampling and query-matching methods.Rule-based methods perform worse than uniform sampling; frame-by-frame retrieval methods still fall short of FRAME-VOYAGER.Source: §4.2 · p. 8
  • Data ablations show additive gains from both datasets, a critical role for filtering, and benefits from ranking better versus worse combinations rather than predicting only the best.Changing the number of combinations used for ranking can adversely affect reward computation.Source: Table 3 · p. 7, §4.2 · p. 9
  • Across frame counts, FRAME-VOYAGER consistently beats uniform sampling and can surpass 16-frame uniform sampling with only 8 frames.The gap narrows as more frames are extracted; the authors note that extra frames may add unnecessary information and inference-model capabilities constrain performance.Source: Figure 3 · p. 8, §4.2 · p. 9
  • Reusing one to two LLM layers improves performance, whereas reusing additional layers leads to gradual decline.The authors attribute the decline to higher-layer features shifting toward language modeling while the LLM remains frozen.Source: §4.2 · p. 9, Figure 4 · p. 9
  • FRAME-VOYAGER significantly improves four of six question categories, including a 4.8% recognition-accuracy gain over uniform sampling, but counting and OCR fluctuate.The sampled case study illustrates how its selected frames track a small black dot’s trajectory and identify it as a virus.Source: Figure 5 · p. 9, §4.2 · p. 10
  • Training on smaller frame sets generalizes to selecting 8 of 128 frames at inference, and pruning reduces data-construction time to 4.4% of the full version while maintaining comparable benchmark performance.The authors report generalization from training on 2 of 16 or 4 of 32 frames to inference on 8 of 128.Source: §4.2 · p. 10

5 CONCLUSIONS

FRAME-VOYAGER is a plug-and-play frame-combination selection method for Video-LLMs, trained through a ranking-based framework and human-free data collection. Experiments report improved VILA performance and results that outperform other frame-selection methods.

5 CONCLUSIONS

  • FRAME-VOYAGER significantly boosts baseline Video-LLMs such as VILA, achieves state-of-the-art results, and outperforms other frame-selection methods.Comprehensive ablations further confirm its effectiveness.Source: §5 · p. 10
  • The method formulates combination optimization as a ranking task and uses a ranking-based reward-learning framework with a human-free data-collection pipeline.Source: §5 · p. 10
  • The authors describe the work as a strong baseline for Video-LLMs and a guide for future research on frame selection and optimization.They also characterize it as an efficient solution for broader video understanding.Source: §5 · p. 10

LIMITATIONS

The limitations concern training-data construction and the plug-in framework; the appendix also describes how several frame-extraction baselines are implemented.

LIMITATIONS

  • Ranking data is generated using VILA-8B only, leaving experiments with more powerful reference Video-LLMs out of scope.Source: LIMITATIONS · p. 10
  • Training combinations primarily cover short videos, while larger frame-combination sets for longer videos could potentially improve extended-video processing.The plug-in reuses backbone parameters without fine-tuning, and its simplified module may not be optimal; integration into pre-training remains unexplored.Source: LIMITATIONS · p. 11

A IMPLEMENTATION DETAILS OF FRAME EXTRACTION BASELINES IN RQ1

  • Baseline implementations use adjacent-frame disparity for SBD, segment keyframes for Katna, VILA-8B visual features for MDF, and grounded-segment sampling for TempGQA.SeViLA uses its localizer with the original hyperparameter settings.Source: A IMPLEMENTATION DETAILS OF FRAME… · p. 18
  • VILA-Embedding ranks frames by pooled visual-text cosine similarity, while CLIP, SigLIP, and InternViT-6B select frames by output logits.Source: A IMPLEMENTATION DETAILS OF FRAME… · p. 18

B QUESTION TYPE ANALYSIS OF THE GENERATED DATASETS

The generated dataset shifts the NextQA question-type distribution toward “what” questions and away from “why is/does” questions. The appendix attributes this filtering pattern to how answer length affects loss calculations.

B QUESTION TYPE ANALYSIS OF THE GENERATED DATASETS

C ANALYSIS OF COMPUTATION COST

FRAME-VOYAGER adds modest parameter and training overhead, alongside higher measured inference latency and memory use than uniform sampling.

C ANALYSIS OF COMPUTATION COST

D RESULTS ON MORE BENCHMARKS

Across MVBench and STAR, FRAME-VOYAGER consistently outperforms uniform sampling, while its EgoSchema improvement is marginal. On Video-MME, expanding the candidate pool improves selection, with limited added information beyond 128 frames.

D RESULTS ON MORE BENCHMARKS

  • FRAME-VOYAGER consistently outperforms uniform sampling on MVBench and STAR, but its improvement on EgoSchema is marginal.The authors report that 44 of 50 sampled EgoSchema questions use “c” to refer to the camera wearer, who may not appear in the video.Source: D RESULTS ON MORE BENCHMARKS · p. 19, Table 4 · p. 19

E A STUDY ON THE NUMBER OF CANDIDATE FRAMES

F DYNAMIC FRAME SELECTION

Dynamic frame selection adapts the number of selected frames to video duration, content domain, and query type. Longer videos, sports content, and more complex queries tend to require more frames.

F DYNAMIC FRAME SELECTION

  • The dynamic-selection analysis uses VILA-8B results to examine required frame counts by video duration, content domain, and query type.The inference experiments use normalized reward thresholds on Video-MME without subtitles, with different frame limits for VILA-8B and LLaVA-OneVision-7B.Source: F DYNAMIC FRAME SELECTION · p. 20
  • Average selected frames rise from 3.72 for short videos to 4.07 for medium and 4.61 for long videos.These counts are reported for short, medium, and long videos, respectively.Source: F DYNAMIC FRAME SELECTION · p. 20
  • Sports videos require 4.43 frames on average, while other video domains require 3.89–4.31.The paper reports that the relationship between frame requirements and video domain is less obvious overall.Source: F DYNAMIC FRAME SELECTION · p. 20
  • Counting and reasoning queries require 5.36 and 4.58 frames on average, compared with 3.01 for perception and 3.84 for recognition.The authors note that counting can depend on multiple frames even when reasoning demands are lower.Source: F DYNAMIC FRAME SELECTION · p. 20

G OPTIMIZATION OF DATA COLLECTION PIPELINE

Two pruning strategies reduce the cost of collecting FRAME-VOYAGER training data. Together, they cut collection time to 4.4% of the original while maintaining comparable benchmark performance.

G OPTIMIZATION OF DATA COLLECTION PIPELINE

  • The pipeline uses dynamic pruning and CLIP-based relevance filtering to reduce computational resource consumption during data collection.The paper presents these as two straightforward efficiency measures.Source: G OPTIMIZATION OF DATA COLLECTION… · pp. 20–22
  • Dynamic pruning based on frame similarity and temporal proximity reduces combinations per sample to 14% on VideoChatGPT and 27% on NextQA.Processing takes about 5 hours on 32 A100 GPUs for VideoChatGPT and about 15 minutes on 8 A100 GPUs for NextQA.Source: G OPTIMIZATION OF DATA COLLECTION… · p. 21
  • CLIP-based question–frame ranking further reduces data-collection time by 60–70% by filtering mostly irrelevant-frame combinations.A few combinations containing low-ranked frames are retained as low-ranking training samples.Source: G OPTIMIZATION OF DATA COLLECTION… · p. 22
  • 4.4% of original data-collection time is sufficient to maintain comparable performance across benchmarks with the optimized pipeline.The comparison uses VILA-8B; Table 10 compares vanilla collection with the two optimized variants.Source: G OPTIMIZATION OF DATA COLLECTION… · p. 22, Table 10 · p. 22

H CASE STUDY ON TRAINING DATA

Case studies show that lower Video-LLM losses tend to identify frame combinations better aligned with question-answer content. On Video-MME examples, FRAME-VOYAGER selects query-relevant frames that capture needed objects and temporal sequences.

H CASE STUDY ON TRAINING DATA

  • Lower-loss NextQA question–answer pairs show stronger visual alignment, while higher-loss frames may contain relevant objects but lack enough information to answer.The authors use these examples to assess the loss-based ranking objective; Figure 8 presents random samples from processed training data.Source: H CASE STUDY ON TRAINING DATA · p. 22, Figure 8 · p. 23

I CASE STUDY ON FRAME COMBINATIONS BY FRAME-VOYAGER

  • For the ingredient query, FRAME-VOYAGER identifies used ingredients, while uniform sampling misses key objects and CLIP misreads special-font text such as “blue food dye”.Figure 10 depicts the frame positions selected by the different methods along the timeline.Source: I CASE STUDY ON FRAME COMBINATIONS BY… · p. 22, Figure 10 · p. 25
  • For the basketball example, FRAME-VOYAGER selects events in order from training to match to podium, while CLIP misses their temporal relationships.Uniform sampling includes many irrelevant background frames; Figure 11 shows frame positions on the timeline.Source: I CASE STUDY ON FRAME COMBINATIONS BY… · p. 22, Figure 11 · p. 26
  • FRAME-VOYAGER selects frames that reflect gradual zoom-in transitions in the correct temporal sequence, unlike uniform sampling or CLIP retrieval.Figure 9 plots the selected frame positions along the video timeline.Source: Figure 9 · p. 24

Related papers

Loading 2410.03226v4…