Source-linked AI summary
Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs
Prakhar Khatri
TL;DR
Long-video models must choose a small subset of thousands of frames, but published comparisons often change several pipeline components at once. This paper holds those components fixed while separately testing selection, spatial compression, and reinvestment across multiple selectors, benchmarks, and answering models. Selection is the largest lever, compression is nearly free at fixed timestamps, and reinvesting its savings in more frames yields additional gains, subject to stated harness and evaluation limits.
Problem
Published selector comparisons change the scorer, prompt boundary, resolution policy, and answering model together, leaving the separate value of selection, compression, and reinvestment unsettled.
Method
The paper uses paired, controlled comparisons that vary selection, per-frame spatial budget, or reinvestment one decision at a time while holding the remaining harness components fixed.
Results
Selection is the largest lever: OMP improves uniform sampling by 5.7 to 11.8 points, compression changes accuracy by at most 0.44 points, and reinvestment returns a further two to three points.
Takeaways & Limitations
The practical ordering is to fix selection first, use resolution as slack, and reinvest saved visual tokens in additional temporal coverage.
Takeaways & Limitations
Generalization is bounded by the controlled harness: equivalence margins were chosen post hoc, some tests are underpowered or descriptive, and encoder-specific mechanisms remain untested.
Abstract
from arXiv · showhide
Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.
1 Introduction
Long-video models must reduce thousands of frames to a small visual-token budget, making frame selection a central bottleneck. This paper isolates selection, compression, and reinvestment in a controlled harness and finds selection is the largest lever, while compression helps only when its savings fund additional frames.
- An hour sampled at one frame per second yields 3,600 images, but long-video systems retain only a small fixed slice, making selection the pipeline’s tightest bottleneck.
- 8 query-selected frames beat 16 uniformly spaced frames by 6.9 points on LongVideoBench’s hour-long bin.The comparison uses the published eight-frame baseline budget and reports p=.0011.
- Halving the per-frame spatial budget changes accuracy by at most 0.44 points at fixed timestamps, while reinvesting the savings in additional keyframes improves long-video QA by two to three points.The equivalence margin was chosen post hoc, and the reinvestment gain is reported as a further improvement.
- OMP outperforms uniform sampling with half as many frames across two LongVideoBench duration bins and stays within one point of LDDR’s stage-1 selector.The matched comparison covers six training-free selectors under one scorer, prompt boundary, frame budget, and answerer harness.
- Replacing LongCLIP with SigLIP changes 67–84% of selected frames yet leaves selector ordering intact within a test that bounds scorer effects above roughly five points.The scorer-swap test bounds large scorer effects rather than establishing their absence.
- A padding bug in the AKS baseline made it select global top-k, and correcting it changed roughly 99.5% of selected frames while moving LVBench accuracy by only 0.07 points.The paper reports cross-arm overlap checks as the diagnostic that exposed the implementation error.
2 Related Work
Related work replaces uniform sampling with query-aware, coverage-based, diverse, learned, jointly resolution-adaptive, or token-level selection. Because those systems change multiple allocation decisions together, the mechanism behind their gains remains unresolved; this paper studies the upstream trade-off separately.
- Training-free selectors combine question relevance, timeline coverage, and within-set diversity using approaches such as MMR, DPPs, and recursive temporal partitioning.These objectives motivate the selector arms compared in the paper.
- Joint frame-selection and resolution systems demonstrate gains, but those gains may reflect better timestamps, better resolution, or simply more temporal coverage.The cited related work does not separate these possible sources.
- Token-reduction methods prune or allocate visual tokens during or after encoding, while AdaAlloc studies global context versus high-resolution local evidence.The paper positions its question upstream of token-level pruning and does not compare against AdaAlloc’s numbers.
3 Method: Three Controlled Interventions
The paper uses paired, matched comparisons to isolate selection, compression, reinvestment, and scorer effects while keeping shared inputs and evaluation conditions fixed. It also tests prompt-boundary sensitivity and explicitly distinguishes reproduced selector schedules from their original pipelines.
- Controlled interventions: Each paired comparison changes only one policy dimension: timestamps for selection, per-frame spatial budget for compression, or frame count after reinvestment.The same benchmark question is answered under both policies, with other conditions held constant.
- Shared scorer: Every selector reads the same cached LongCLIP stem embeddings, preventing encoder quality from confounding the selection comparison.Videos are decoded at 1 fps, and answer options are excluded from the selection text unless explicitly tested.
- Prompt boundary: The prompt boundary is consequential: using question-and-options text changes 41.9% of top-k selections and 53.3% of OMP selections on LongVideoBench-600 s.With fused queries, OMP reaches .6699 versus .6311 for the stem, while top-k reaches .6408 versus .6068.
- Selection rules: The matched selector comparison includes uniform sampling, cosine top-k, AKS, FOCUS⋆, OMP, and LDDR-select, all evaluated under the shared LongCLIP stem scorer.FOCUS⋆ replays the published temporal schedule rather than the original budgeted online ITM process, while LDDR-select is only its stage-1 Linear-DPP selector.
- Selection rules: OMP selects frames greedily by correlation with the current residual, then projects away directions already covered by earlier selections.The paper adopts the 1993 sparse-approximation algorithm unmodified and treats its embedding-space interpretation as an empirical question.
- Compression and reinvestment: Compression uses fixed timestamps and controlled resizing, while reinvestment compares eight full-resolution frames with sixteen compressed frames at empirically matched token cost.The main fixed-timestamp arm targets a mean spatial fraction of about 0.53; the reinvestment arm audits actual Qwen smart-resize token counts.
4 Evaluation Setup
The evaluation spans three long-video benchmarks and uses paired accuracy comparisons under matched inputs. Exact McNemar tests assess differences, while TOST intervals support claims that compression arms are interchangeable within specified margins.
- Datasets: Three benchmarks are evaluated: LongVideoBench (n=1337), Video-MME (n=2700), and LVBench (n=1549).LongVideoBench is additionally divided into 15, 60, 600, and 3600 s duration bins.
- Experimental unit: The evaluation grid prioritizes paired accuracy differences under matched inputs rather than absolute leaderboard position.Each question is answered twice, once under each input policy, with the same surrounding conditions.
- Statistics: Exact two-sided McNemar tests compare paired correctness outcomes, while TOST with 90% confidence intervals bounds claimed interchangeability.The ±2, ±3, and ±4 percentage-point margins were chosen after seeing the runs and were not preregistered; tests are uncorrected for multiplicity.
5 Results
Across controlled comparisons, selection matters most: query-aware rules can outperform uniform sampling even with half as many frames, while compression is nearly free and reinvestment recovers further accuracy. These conclusions are bounded by scorer, benchmark, and harness effects.
- 5.1 Choosing frames beats having more of them: 6.9 points: OMP with eight frames reaches .5461 versus .4770 for uniform sampling with sixteen on LongVideoBench 3600 s.At 600 s, OMP with sixteen frames gains 5.3 points over uniform sampling with thirty-two frames.
- 5.1 Choosing frames beats having more of them: 5.69 to 11.81 points: OMP improves on uniform sampling across LongVideoBench, Video-MME, and LVBench.LDDR-select stays within a point of OMP on all three benchmarks.
- 5.1 Choosing frames beats having more of them: 7.8 points at 600 s and 7.5 at 3600 s: OMP's gain over uniform appears when the 1 fps candidate pool exceeds the eight-frame budget.At 15 s, uniform, top-k, and OMP have identical accuracy (.7249), so the effect switches on with pool size rather than duration.
- 5.2 The ranking survives a scorer swap: The scorer-swap evidence preserves the selector ordering but does not establish scorer-invariance in general.LongCLIP and SigLIP share a contrastive image–text objective, so scorers built on different principles could still reorder the table.
- 5.3 Half the pixels change almost nothing: 0.44 points: fixed-timestamp spatial compression changes accuracy by at most that amount across the three Qwen benchmark aggregates.On pooled LongVideoBench long bins, the compressed OMP arm is 0.72 points higher, with a 90% interval of [−0.60, +2.03] fitting within ±3 points.
- 5.3 Half the pixels change almost nothing: 1.84 points: a reconstruction of LDDR's stage-2 Group-DPP importance improves on a flat split in pooled long bins.By contrast, the residual-proportional schedule does not beat a flat split at the same mean budget, and no equivalence test was run.
- 5.4 Spending the savings on more frames: 2.24 points: spending recovered spatial budget on sixteen compressed rather than eight full-resolution frames improves LongVideoBench overall.The pooled 600 s and 3600 s bins improve by 2.36 points, while selectorspecificity remains open at roughly the one-point scale.
6 Discussion
Under controlled comparisons, selection is the largest allocation lever, while compression matters mainly when its savings purchase additional temporal coverage. The discussion also emphasizes implementation audits, harness sensitivity, and decomposed reporting.
- Compression is close to free but buys little alone; reinvesting its savings in temporal coverage returns a further two to three points.
- A corrected AKS implementation changed roughly 99.5% of selected frames yet moved aggregate accuracy by only 0.07 points on LVBench.
- Pairwise overlap checks exposed the AKS bug, showing why aggregate accuracy and per-arm plausibility checks can miss implementation errors.
- 0.07 to 3.74 points separate two matched harnesses running the same rules at the same budget, making cross-paper selector deltas fragile.
- Future methods should report separate timestamp, resolution, and frame-count interventions alongside joint results.
7 Limitations
The paper’s conclusions are bounded by reproduction choices, post-hoc equivalence margins, encoder-specific mechanism evidence, compute-environment differences, and incomplete evaluation coverage.
- FOCUS⋆ and LDDR-select are controlled reproductions rather than reproductions of the original full systems.
- Equivalence margins were chosen after inspecting the data, so the ±3-point result is descriptive rather than confirmatory.
- Mechanism evidence comes from frozen LongCLIP embeddings, while BLIP-ITM, MLLM-attention, and subtitle-aware selectors remain untested.
- 0.71 points lower is the corresponding OMP reading on the second compute stack in the 3600 s bin, limiting comparability of absolute accuracies across environments.
- Subtitles are disabled, GPT-5-mini uses pixel-ratio accounting, some selector results lack per-item records, and MDP3 and Q-Frame are not run.
8 Conclusion
In the matched harness, selecting relevant frames matters most, compression provides slack, and reinvestment of that slack into temporal coverage improves results. Generalization depends on video length, benchmark, and answering model.
- Selection is the largest allocation decision: eight well-chosen frames beat sixteen uniformly spaced ones in LongVideoBench’s hour-long bin.
- OMP improves over uniform sampling by 5.7 to 11.8 points and remains within one point of a purpose-built modern selector.
- Roughly half the per-frame spatial budget can be removed with little observed change, while reinvesting those tokens in additional keyframes returns two to three points.
- The reported gains depend on video length, benchmark, and the model doing the reading.
9 Ethical Considerations
The experiments use existing public benchmarks and models without collecting new human subjects, personal data, videos, or annotations.
- The study collects no new human subjects, personal data, videos, or annotations.
- Benchmark videos remain governed by their original licenses and are not redistributed with the released artifacts.
A FOCUS⋆schedule replay
The FOCUS⋆ replay evaluates its temporal schedule using a shared, precomputed LongCLIP scorer rather than the native budgeted ITM-scoring process. Its starred results therefore isolate schedule behavior under matched scoring and do not assess native FOCUS efficiency or accuracy.
- FOCUS⋆ replays the original temporal clip schedule while exposing precomputed LongCLIP scores to the bandit.The replay preserves default hyperparameters and tests the schedule under a matched scorer.
- FOCUS⋆ does not support conclusions about native FOCUS under its ITM protocol.The star marks the boundary created by revealing dense LongCLIP scores instead of budgeting scoring itself.
- The replay uses 16 s clips, three coarse pulls per clip, α=0.25, and a pull budget equal to half the candidate pool.Approximately k/4 clips survive, with unobserved rewards interpolated from the nearest observed frame and a fixed random seed per item.
B Question categories do not define a stable selector regime
The study tests whether temporally referred question categories require a distinct selector regime, but finds no significant category interaction in either long-duration bin. Selection variants remain close to OMP, with no tested variant significantly outperforming it.
- Question categories: The T* group compares OMP with uniform sampling on temporally referred categories against all remaining LongVideoBench categories.The data are tested without relabeling, using the benchmark’s existing question-category tags.
- Question categories: The category interaction is not significant in either long bin, despite percentage-point gains differing by duration.The temporal slice appears resistant to selection at 600 s and responsive at 3600 s, but which slice wins reverses between bins.
- Selector variants: Most orthogonalization and diversity variants remain within a few points of OMP, while MMR and query-blind DPP lose.No tested variant significantly beats OMP in the reported comparison.
- Question categories: The apparent temporal-category instability may reflect vision-only selection when many T* questions refer to subtitles or speech.Subtitles are disabled, so the proposed explanation remains plausible rather than established.
C Selector variants
Within the tested scorer geometry and budget range, changing relevance–diversity rules produces less variation than replacing uniform sampling or reallocating pixels into frames. Failure review suggests a recurring but non-prevalence claim: irrelevant distinctive content can consume OMP picks.
- Selector variants: MMR underperforms, query-blind DPP collapses toward uniform sampling, and other tested variants cluster around OMP.No tested DPP variant significantly exceeds OMP in either long bin.
- Evaluation boundary: The sweep is internally paired within a secondary compute environment and is not comparable with Tables 2–5.This scope boundary applies to cross-table interpretation.
- Selector variants: Within this scorer geometry and budget range, relevance–diversity adjustments change performance less than replacing uniform sampling or reallocating pixels into frames.The result is consistent with query-directed signal being largely exhausted after roughly five picks.
- Failure analysis: Failure review inspected 41 cases in the 600 s bin and 52 in the 3600 s bin across broad category coverage.Cases were selected to include OMP-only and shared failures, so the sample was not random.
- Failure analysis: Visually distinctive but irrelevant content frequently consumed several picks in both reviewed passes.Examples include unrelated chapters, separate storylines, title cards, subscribe cards, and dark transitions; the conditioned sample cannot establish dominance or prevalence.