PaperlayerView exact revision on arXiv

Source-linked AI summary

BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding

Shuming Liu, Chen Zhao, Tianqi Xu, Bernard Ghanem

arXiv:2503.21483v1cs.CV

TL;DR

Long-form video understanding is constrained by limited context and irrelevant or redundant frame selection, while existing adaptive methods require additional training. BOLT evaluates multi-source retrieval and applies query-aware frame selection only at test time, with inverse transform sampling producing the strongest reported gains, including Video-MME accuracy from 53.8% to 56.1% and MLVU accuracy from 58.9% to 63.4%.

  • Problem

    Long-form video analysis is limited by language-model context length, while uniform sampling can allocate resources to irrelevant content and existing adaptive methods require additional training.

  • Method

    BOLT evaluates multi-source retrieval and explores query-aware frame selection strategies, including inverse transform sampling, applied at test time without retraining VLMs.

  • Results

    BOLT consistently improves performance across five benchmarks, including Video-MME accuracy from 53.8% to 56.1% and MLVU accuracy from 58.9% to 63.4%.

  • Takeaways & Limitations

    Inverse transform sampling provides robust gains over uniform sampling, especially in retrieval-based evaluation, where accuracy increases from 47.78% to 53.35%.

  • Takeaways & Limitations

    BOLT relies on accurate query-frame similarity measures, which may not reflect true relevance for reasoning tasks, while grouping-based selection can overlook temporal context and retain redundancy.

Abstract

from arXiv · show

Large video-language models (VLMs) have demonstrated promising progress in various video understanding tasks. However, their effectiveness in long-form video analysis is constrained by limited context windows. Traditional approaches, such as uniform frame sampling, often inevitably allocate resources to irrelevant content, diminishing their effectiveness in real-world scenarios. In this paper, we introduce BOLT, a method to BOost Large VLMs without additional Training through a comprehensive study of frame selection strategies. First, to enable a more realistic evaluation of VLMs in long-form video understanding, we propose a multi-source retrieval evaluation setting. Our findings reveal that uniform sampling performs poorly in noisy contexts, underscoring the importance of selecting the right frames. Second, we explore several frame selection strategies based on query-frame similarity and analyze their effectiveness at inference time. Our results show that inverse transform sampling yields the most significant performance improvement, increasing accuracy on the Video-MME benchmark from 53.8% to 56.1% and MLVU benchmark from 58.9% to 63.4%. Our code is available at https://github.com/sming256/BOLT.

1. Introduction

Long-form video understanding is constrained by limited context windows and ineffective uniform sampling, especially in noisy real-world videos. BOLT studies training-free, query-aware frame selection and finds inverse transform sampling consistently improves performance.

  • 20 million tokens would be required for an hour-long video at 30 FPS, making video token reduction essential under context-length and computational constraints.
  • Uniform frame sampling can allocate resources to irrelevant or distracting content and fail in complex real-world scenarios.
  • Adaptive similarity-based methods such as Goldfish, LongVU, and Frame-Voyager select relevant frames but require additional training and substantial computational resources.Frame-Voyager requires 32 H100 GPUs for training.
  • BOLT introduces query-aware top-K selection, watershed grouping, and inverse transform sampling, all applied at test time without retraining VLMs.
  • BOLT achieves consistent improvements on five benchmarks, including Video-MME accuracy from 53.8% to 56.1% and MLVU accuracy from 58.9% to 63.4%.

2. Related Works

Related work addresses large VLMs’ video-token and context limitations through compression and adaptive frame selection. BOLT differs by improving existing VLMs with training-free frame selection at inference time.

  • Large VLMs face substantial video-processing challenges from many frames, limited language-model context, and the “lost in the middle” problem.
  • Uniform sampling reduces tokens straightforwardly but ignores frame importance and may omit crucial information from complex or long videos.
  • Token Merging: Prior token-reduction approaches include merging information within or across frames to create compact representations while preserving visual details.
  • Query-Based Frame Sampling: Similarity-based methods retrieve query-relevant frames or clips, while some approaches learn frame-selection strategies through additional training.
  • Query-Based Frame Sampling: BOLT proposes a training-free frame-selection strategy that enhances existing large VLMs at inference time.

3. Revisiting Large VLMs

Large VLMs select a limited number of video frames, encode them, combine their embeddings with question text, and generate answers within a fixed context budget. The section frames frame selection as central to this pipeline and evaluates the importance of using question-relevant segments.

  • N ≪ T frames are selected from a high-frame-rate video before visual encoding, with common implementations often using 16 or 32 frames.
  • Each selected frame is processed by a vision encoder E_v to produce visual embeddings v_i ∈ R^M×C, where C is embedding dimension and M is tokens per frame.
  • Visual embeddings are projected into the language space, concatenated with textual question embeddings E_t(Q), and processed by an LLM to generate answer A autoregressively.
  • The context-length limit L constrains the combined number of visual and textual tokens supplied to the language model.
  • Because the model must select frames within a limited context window, frame selection is crucial for effective video understanding.

4. BOLT

BOLT studies training-free frame selection for long-form video understanding, evaluating whether query-relevant frames improve VLM performance in noisy, extended videos. It introduces multi-source retrieval evaluation and finds inverse transform sampling to be the strongest tested strategy.

  • 4. BOLT: Ground-truth segment-only sampling helps VLMs focus on informative visual features, while irrelevant frames reduce answer similarity.Experiments use InternVL2-8B and LLaVA-NeXT-Video-7B with 8 sampled frames and average sentence similarity.
  • 4. BOLT: The multi-source retrieval setting concatenates distinct, similarly timed videos with different visual content to create longer, noisier evaluation videos.The setting is designed to approximate real-world extended sequences and impose more demanding retrieval and reasoning requirements.
  • 4. BOLT: 56.85% to 52.11%: uniform-sampling accuracy declines under multi-source retrieval even with 64 frames; 16 frames falls to 47.78%.These results indicate that adding query-irrelevant content substantially challenges uniform sampling.
  • 4. BOLT: Top-K selection retrieves high-similarity frames but can produce redundant views and miss diverse context needed for temporal dependencies and causal reasoning.Watershed grouping improves diversity by clustering adjacent frames, but may overlook broader temporal context and retain redundancy.
  • 4. BOLT: Inverse transform sampling combines query relevance, frame diversity, and surrounding context without retraining the VLM.It constructs a query-guided probability distribution from normalized frame-query similarity scores and is applied at test time.
  • 4. BOLT: 2.3% on Video-MME and 4.5% on MLVU: inverse transform sampling improves over uniform sampling without additional training.The strategy consistently outperforms alternative approaches during inference.

5. Experiments

Experiments evaluate BOLT across standard and noisy long-form video benchmarks, showing that query-aware, training-free frame selection—especially inverse transform sampling—outperforms uniform sampling. Ablations examine generalization, retrieval noise, similarity measures, hyperparameter sensitivity, and frame-selection behavior.

  • Benchmark Setup: The evaluation covers five VQA benchmarks, with Video-MME, MLVU, and LongVideoBench representing longer videos than NextQA and EgoSchema.Long-video benchmarks span a few minutes to around two hours, while NextQA and EgoSchema average less than two minutes.
  • Standard Benchmarks: 56.1% versus 53.8% accuracy on Video-MME and 63.4% versus 58.9% on MLVU shows ITS improves LLaVA-OneVision with 8 frames.The comparisons use the baseline and ITS values reported for each benchmark.
  • Generalization: BOLT improves off-the-shelf VLM accuracy by approximately 1% to 3% on Video-MME, including LLaVA-Video’s increase from 56.0% to 58.6%.The approach generalizes beyond LLaVA-OneVision while using inverse transform sampling.
  • Multi-Source Retrieval: In multi-source retrieval, irrelevant background frames reduce LLaVA-OneVision accuracy from 56.9% to 47.8% with 16 frames, while BOLT raises it from 47.8% to 53.4%.Increasing the budget to 64 frames only partially recovers performance, reaching 52.1%.
  • Similarity Measures: Direct image encoders outperform caption-based similarity, CLIP outperforms SigLIP, and averaging CLIP and SigLIP scores performs best overall.The caption-based approach is weakened by noisy captions.
  • Hyperparameter Analysis: ITS consistently surpasses uniform sampling when α ranges from 2 to 3.5, with only minor performance variation within that interval.The analysis indicates limited sensitivity to the exact sharpness value.
  • Frame-Selection Behavior: ITS selects high-similarity frames while also including local peaks as contextual information, preserving frame diversity for reasoning.Figure 3 visualizes similarity scores across the video and the frames selected by inverse transform sampling.

6. Conclusion

The conclusion presents BOLT as a training-free approach that uses temporal frame selection, especially inverse transform sampling, to improve long-form VLM understanding without fine-tuning. It also identifies dependence on accurate query-frame similarity as a limitation.

  • Conclusion: BOLT enhances large VLMs for long-form video understanding through training-free temporal frame selection focused on query-relevant frames.The method is designed to improve visual question-answering performance without additional fine-tuning.
  • Limitations: BOLT’s effectiveness relies on accurate query-frame similarity measures, which may not reflect true relevance for reasoning-intensive or multi-round VQA tasks.The paper suggests developing more robust similarity metrics for varied video content and query contexts.

Supplementary Material

The supplementary material adds experimental results and analyses covering subtitles, inference costs, and visualization examples.

  • Supplementary Scope: Supplementary sections provide Video-MME results with subtitles, inference-cost analysis, and additional visualization examples.These materials appear in Sections A, B, and C, respectively.

A. Additional Results on Video-MME

Additional Video-MME experiments show that BOLT improves performance when subtitles are included, across different frame budgets and video durations.

  • Subtitle Setting: 61.0% versus 58.7% accuracy with 8 input frames demonstrates BOLT’s improvement on Video-MME with subtitles.Performance also improves across short, medium, and long videos.

B. Analysis of Inference Cost

BOLT adds test-time frame encoding and selection to improve long-video VLM inputs without retraining. This keeps memory nearly unchanged but increases per-sample inference time by approximately 90%, mainly because of CLIP encoding.

  • Approximately 90%: total inference time per sample increases, primarily because of CLIP feature encoding.Inverse transform sampling itself is highly efficient.
  • Nearly the same GPU memory: BOLT requires about as much memory as uniform-sampling baselines.
  • One A100 GPU, LLaVA-OneVision-7B, 16 frames, and CLIP-L/14 define the Table 11 inference-time evaluation setup.
  • Basic CLIP matching avoids auxiliary alignment models, object detectors, and OCR models that would increase computational overhead.The frame-selection method remains orthogonal to these approaches.

C. Additional Qualitative Results

The qualitative results show that inverse transform sampling selects high-similarity frames while retaining some background context. When similarity is nearly uniform across a video, the method approximates uniform sampling because most clips may contribute equally.

  • Blue curves show visual-query similarity across the video, while red lines mark the frames selected by inverse transform sampling.
  • High visual-query similarity: the proposed method selects corresponding frames while preserving some background elements for contextual information.The retained background helps maintain context required for accurate video understanding.
  • Inverse transform sampling simplifies to uniform sampling when visual-query similarity scores remain similar across the video.
  • Figure 5 provides additional visualization results for the frame-selection behavior.
  • Approximately uniform sampling: when similarity scores remain similar throughout the video, inverse transform sampling selects frames approximately uniformly.The cumulative distribution function becomes approximately linear in this case.
  • Most clips may contribute equally to answering the question when visual-query similarity remains relatively consistent across the video.
Loading 2503.21483v1…