PaperlayerView exact revision on arXiv

Source-linked AI summary

Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs

Prakhar Khatri

arXiv:2609.03820v1cs.CVcs.CL

TL;DR

Long-video systems retain only a small fixed frame slice, but the choice of which frames to keep is often treated as preprocessing. This paper isolates selection, compression, and reinvestment in paired comparisons; selection is the largest lever, with eight query-selected frames beating sixteen uniformly spaced ones by 6.9 points on LongVideoBench’s hour-long videos.

  • Problem

    Long-video systems retain only a small fixed frame slice, but the frame-selection rule is often treated as preprocessing rather than a decision to evaluate.

    Source: §1 · p. 1

  • Method

    The study uses paired comparisons in one controlled harness to vary frame selection, spatial compression, and reinvestment while holding other inputs fixed.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across the allocation decisions, selection is the largest lever; eight query-selected frames beat sixteen uniformly spaced frames by 6.9 points on LongVideoBench’s hour-long videos.

    Source: §1 · p. 1, §6 · p. 11

  • Takeaways & Limitations

    The practical ordering is to prioritize selection, use resolution as slack, and reinvest compression savings in temporal coverage.

    Source: §6 · p. 11

    Mechanism evidence is limited to frozen LongCLIP embeddings, and BLIP-ITM, MLLM-attention, and subtitle-aware selectors remain untested.

    Source: §7 · p. 13

Abstract

from arXiv · show

Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.

1 Introduction

Because long-video models retain only a small slice of a video’s frames, this study isolates selection, compression, and reinvestment in a matched harness. It finds selection is the largest lever, while compression yields accuracy gains only when its savings are reinvested.

1 Introduction

  • 6.9 points: on hour-long LongVideoBench videos, eight query-selected frames beat sixteen uniformly spaced frames (p=.0011).The same pattern holds for ten-minute videos; the selection stage’s cost is accounted for separately.Source: §1 · p. 1
  • The study changes one decision at a time—timestamps, per-frame spatial budget, or spending recovered budget on more timestamps—while holding the other pipeline components fixed.Paired comparisons on the same questions attribute differences to the intervention; a scorer swap checks whether selector ordering depends on the frozen encoder.Source: §1 · p. 2
  • Halving each frame’s spatial budget at fixed timestamps changes accuracy by at most 0.44 points, while reinvesting the savings in more keyframes adds two to three points.The pooled-long-video equivalence interval is post-hoc ±3 points, and the intervention ranking synthesizes separate contrasts rather than a factorial design.Source: §1 · p. 2
  • OMP improves on uniform sampling by 5.7 to 11.8 points across benchmarks and stays within one point of LDDR’s stage-1 selector.The comparison matches six training-free selectors while fixing the scorer, prompt boundary, frame budget, and answerer harness; OMP is adopted unmodified and untuned.Source: §1 · p. 2
  • Replacing LongCLIP with SigLIP changes 67–84% of selected frames yet leaves selector ordering intact, with the test bounding scorer effects only above roughly five points.The scorer swap covers one bin, and the equivalence result covers one pooled setting.Source: §1 · pp. 2–3
  • A padding bug made the AKS port select global top-k for every k < 32; correcting it changed roughly 99.5% of selected frames but only 0.07 points of LVBench accuracy.A cross-arm overlap check caught the implementation error.Source: §1 · p. 2

2 Related Work

Related work spans query-aware frame selection, joint frame-resolution allocation, and visual-token reduction; this study isolates the upstream choice between sharper retained frames and additional moments.

Query-aware frame selection

  • Training-free query-aware selectors trade off question relevance, timeline coverage, and within-set diversity, but comparisons are confounded by differing scorers, candidate pools, training requirements, and frame budgets.The literature includes classical subset-selection rules and newer bandit-, DPP-, segment-, and uncertainty-based methods.Source: §2 · p. 3

Selection with spatial adaptation

  • Q-Frame (Zhang et al., 2025a), LDDR (Chen et al., 2026a), and DAFS (Wang et al., 2026b) jointly allocate frames and resolution, leaving gains' sources among timestamps, resolution, and temporal coverage unresolved.The cited work does not separate these three possible sources of improvement.Source: §2 · p. 3

Visual-token reduction

  • Token-level reducers cut cost during or after visual encoding, while AdaAlloc asks how to divide a fixed budget between global context and high-resolution local evidence.At writing, AdaAlloc had no public preprint or code, so the authors describe it from its project-page abstract and make no comparison with their results; their upstream question is whether answers benefit more from sharper retained frames or additional moments.Source: §2 · p. 3

3 Method: Three Controlled Interventions

The study isolates frame selection, spatial compression, and reinvestment by comparing paired answers while holding other factors fixed. It also tests how scorer input and spatial-budget choices affect those comparisons.

3.1 The experimental unit

  • Each benchmark question is answered under two input policies, with selection, compression, and reinvestment changing one decision at a time.Selection varies timestamps while fixing frame count, scorer, and resolution; compression fixes timestamps and changes spatial budget; reinvestment spends compression savings on more timestamps.Source: §3.1 · p. 3

3.2 One scorer, shared by every rule

  • On LongVideoBench-600 s, fused-query OMP scored .6699 versus .6311 for stem-only scoring (+3.88 points), showing that scorer prompt boundaries can shift accuracy.Fused-query top-k scored .6408 versus .6068 for stem-only (+3.40 points); on the 3600 s bin, OMP scored .5691 versus .5461 (+2.30 points, n=564). Irrelevant substituted options recovered 56% of the 600 s gain, while the item’s own options added a further 1.70 points beyond that, not significantly (p=.42); the study therefore uses stem-only scoring as a conservative configuration.Source: §3.2 · pp. 4–5

3.3 Selection rules

  • OMP’s residual projection suppresses directions already covered by earlier picks, and the matched comparison evaluates it alongside five other selection rules.Those rules are uniform sampling, cosine top-k, AKS, FOCUS⋆, and LDDR-select; all use LongCLIP stem scores, though that scorer is native only to LDDR-select, and FOCUS⋆ replays a published schedule on dense scores.Source: §3.3 · p. 5

3.4 Spatial budget and reinvestment

  • For fixed timestamps, D@53 targets a mean spatial fraction of about 0.53; reinvestment instead compares eight full-resolution frames with sixteen frames under an approximately 50% per-frame cap.Controls test a flat pixel split and LDDR’s Group-DPP importance on the same OMP-8 timestamps. The sixteen-frame arm used 0.996 and 0.984 times the tokens of the eight-frame arm in the LongVideoBench 600 s and 3600 s bins, respectively; parity was not separately audited for Video-MME or LVBench.Source: §3.4 · p. 5

4 Evaluation Setup

The evaluation compares frame-selection policies on three video benchmarks using matched inputs and paired accuracy differences. It uses paired tests for accuracy and equivalence tests when claiming compression arms are interchangeable.

4 Evaluation Setup

  • Evaluation covers LongVideoBench (n=1337), Video-MME (n=2700), and LVBench (n=1549), with LongVideoBench stratified into four duration bins.The complete official LongVideoBench validation set is used (Wu et al., 2024); Video-MME (Fu et al., 2025) and LVBench (Wang et al., 2024) provide additional benchmarks. Qwen3-VL-8B-Instruct (Qwen Team, 2025) is the primary answerer, run via lmms-eval (EvolvingLMMs-Lab, 2024) with greedy decoding, temperature zero, and subtitles disabled; InternVL3-2B and InternVL3-8B (Zhu et al., 2025a) provide a second model family, and GPT-5-mini checks selected contrasts on the same saved timestamps.Source: §4 · p. 5
  • The evaluation grid emphasizes paired accuracy differences under matched inputs, not absolute leaderboard position.Source: Table 1 · p. 6
  • Accuracy comparisons use paired McNemar tests; interchangeability claims use TOST with 90% confidence intervals rather than relying on nonsignificance.Equivalence margins of ±2, ±3, and ±4 percentage points were chosen after seeing the runs, so the study reports the narrowest supported margin and the adjacent unsupported margin; tests are contrast-specific and uncorrected for multiplicity.Source: §4 · p. 6

5 Results

Across these results, query-based frame selection often outperforms using more uniform frames, while spatial compression costs little and can fund accuracy gains when its savings buy more frames. The size and consistency of these effects depend on the benchmark and answering model, and the proposed explanations for OMP’s later picks remain hypotheses.

5.1 Choosing frames beats having more of them

  • OMP improves on uniform sampling by 5.69 points on LongVideoBench, 5.85 on Video-MME, and 11.81 on LVBench; at 3600 s, eight OMP frames beat sixteen uniform frames by 6.9 points.The eight-versus-sixteen-frame LongVideoBench contrast is .5461 versus .4770 (p=.0011); the advantage appears when the candidate pool exceeds the budget, not on 15 s clips.Source: §5.1 · p. 6, Table 2 · p. 7

5.2 The ranking survives a scorer swap

  • With LongCLIP and SigLIP, uniform sampling is beaten by top-k and top-k by OMP on LongVideoBench-600 s, but the test bounds large scorer effects rather than proving scorer-invariance.The scorers select substantially different frames, yet OMP beats uniform significantly under both (p=.0024 and p=.00026); only two related contrastive encoders were tested, and the OMP scorer-effect interval spans about [−3.6, +5.6] points.Source: §5.2 · pp. 7–8, Table 3 · p. 7

5.3 Half the pixels change almost nothing

  • Across three Qwen benchmark aggregates, compression shifts accuracy by at most 0.44 points; residual-proportional allocation adds no detectable gain, while reconstructed Group-DPP importance adds 1.84 points on pooled long bins.The Group-DPP contrast is indicative: it is a same-environment reconstruction, and its 3600 s component is not individually significant. The residual-proportional versus flat-split contrast was not tested for equivalence.Source: §5.3 · p. 8, Figure 3 · p. 8

5.4 Spending the savings on more frames

  • Sixteen compressed frames improve LongVideoBench by 2.24 points overall, LVBench by 3.04, Video-MME by 1.56, and GPT-5-mini on LVBench by 2.39; equal-or-lower measured token cost is established only for LongVideoBench.Source: §5.4 · p. 9, Table 5 · p. 9

5.5 What the later picks actually add

  • OMP’s residual norm barely changes after eight picks while its correlation with the selected frame falls from .233 to .004; the authors hypothesize later picks add coverage, not query direction.The account is observational, and mediation was not tested. An exploratory audit found off-topic novelty in 20 of 41 inspected 600 s failures and 35 of 52 inspected 3600 s failures; dense top-k could also miss another required scene.Source: §5.5 · pp. 9–10, Figure 5 · p. 10

5.6 The same frames do not help every model equally

  • With InternVL3, OMP’s selection gains replicate on LongVideoBench and LVBench, but Video-MME benefits vary by model size and duration; 8B gains 3.38 points when 8-frame sampling becomes 24-frame sampling at fixed tile budget.InternVL3-8B also holds pooled LongVideoBench long-bin accuracy through a threefold tile reduction (−0.10 points, p=1.000); selection and allocation effects vary by answerer–benchmark pair.Source: §5.6 · p. 10, Table 6 · p. 10

6 Discussion

The discussion argues that selection is the largest allocation lever, compression helps accuracy only when its savings are reinvested in more temporal coverage, and controlled comparisons expose substantial selector, harness, and implementation effects. It also cautions that selector performance depends on the benchmark, duration, and answerer, and recommends decomposing joint allocation systems into separate interventions.

Allocation priorities

  • Spending compression savings on twice as many frames returns two to three points, while compression alone changes few predictions and costs at most 0.44 points.The practical order is to improve frame selection first, treat resolution as a slack variable, and reinvest the savings; compression may still help latency or memory without reinvestment.Source: §6 · p. 11

Selector comparisons

  • On LVBench, the uniform-to-top-k gap is 8.65 points versus a 4.06-point between-rule span; on LongVideoBench, the gaps are 3.74 and 4.04 points, respectively.On Video-MME, the between-rule span is 5.18 points versus 0.67 for top-k over uniform; OMP leads AKS, FOCUS⋆, and top-k on all three benchmarks and is within a point of LDDR’s stage-1 selector. The discussion recommends shared scoring stages and a classical pursuit or greedy baseline when evaluating new rules.Source: §6 · p. 11

Harness disagreement

  • 0.07 to 3.74 points separate the gains in two matched-condition harnesses running the same published rules, showing cross-paper selector deltas of this size are fragile.The harnesses’ uniform baselines differ, including 28.08 against 34.54 on LVBench; the authors offer a lower floor as a hypothesis, not an explanation. They treat Table 7 as a measurement of harness disagreement, not a reproduction verdict.Source: Table 7 · p. 11, §6 · pp. 11–12

AKS implementation

  • Correcting the AKS depth changes roughly 99.5% of selected frames, while accuracy shifts −1.05 points on LongVideoBench, +0.07 on LVBench, and +2.74 on Video-MME.The earlier padding branch silently filled all eight frame slots with global cosine top-k; an overlap check exposed 98.5% agreement with top-k. The corrected row reproduces the published Video-MME gain within 0.07 points, versus a 2.81-point gap under the bug.Source: §6 · pp. 11–12

Decomposing allocation systems

  • The controls find no value in simple priority-proportional resolution, a positive effect from reconstructed LDDR-style importance, and the clearest downstream gain from temporal reinvestment.Because systems such as Q-Frame, LDDR, and DAFS combine causal routes, the authors recommend reporting timestamp, resolution, and frame-count interventions alongside end-to-end results.Source: §6 · p. 12

OMP mechanism

  • OMP’s relevance-plus-decorrelation behavior can extend multi-scene coverage after the text residual flattens, but unconstrained diversity can reward irrelevant novelty.The discussion identifies relevance floors, event-boundary constraints, and scorers with a smaller modality gap as next interventions, and proposes residual traces as a cheap check for query signal.Source: §6 · p. 12

Scope of conclusions

  • InternVL3-8B is nearly insensitive to OMP on medium and long Video-MME yet gains over ten points on LVBench; OMP’s advantage over uniform is exactly zero at 15 s.The discussion cautions against generalizing keyframe-selection claims from a single benchmark, duration, or answerer.Source: §6 · p. 12

7 Limitations

The limitations constrain how the selector comparisons should be interpreted: some methods are controlled reproductions, statistical evidence is partly exploratory, mechanism claims are encoder-specific, and secondary results use a separate compute stack. Evaluation also omits subtitles and two comparison methods, with some model-specific accounting and recordkeeping limitations.

7 Limitations

  • FOCUS⋆ tests its published schedule with dense LongCLIP scores, not its budgeted ITM scorer; LDDR-select runs stage 1 only, with stage 2 reconstructed rather than run from the authors’ code.Neither row is a reproduction result for the original method.Source: §7 · pp. 12–13
  • Qwen’s pooled LongVideoBench 600 s and 3600 s examples clear the post hoc ±3-point margin but miss ±2; tests are uncorrected, and several comparisons lack confirmatory evidence.The margin was chosen after inspecting the interval; confirmatory equivalence would require a prespecified margin and fresh replication. Video-MME compression has no paired test, LVBench differences are descriptive, and the selector-by-budget interaction is underpowered for effects near one point.Source: §7 · p. 13
  • Mechanism evidence comes from frozen LongCLIP embeddings, and the model-assisted failure audit sampled selected failures, limiting generalization and population-rate estimates.Selector ordering survives a SigLIP swap, but BLIP-ITM, MLLM-attention, and subtitle-aware scorers remain untested; the modality-gap explanation is specific to this embedding space.Source: §7 · p. 13
  • On the second stack, the uniform eight-frame full-resolution arm is 0.36 points higher in the 3600 s bin and identical in the 600 s bin, while OMP is 0.71 points lower.Every arm of each contrast, including a rerun of the full-resolution baseline, ran within that stack, so paired differences are valid; absolute accuracies are not comparable to Tables 2–5.Source: §7 · p. 13
  • Subtitles are disabled, GPT-5-mini compression uses pixel ratio rather than Qwen token accounting, and some GPT selector results survive only as aggregates.MDP3 and Q-Frame were omitted because neither has a published number at the exact comparison cell for validation; the authors judged two more unvalidated rows worse than a stated gap, noting their respective overlap with DPP behavior and composite selection-resolution evaluation.Source: §7 · p. 13

8 Conclusion

The conclusion recommends selecting relevant frames first, compressing them second, and reinvesting the savings to see more of the video, while noting that benefits depend on video length, benchmark, and answering model.

8 Conclusion

  • Selection was the largest allocation decision: eight well-chosen frames beat sixteen uniformly spaced ones in LongVideoBench’s hour-long bin, and unmodified OMP outperformed uniform sampling by 5.7 to 11.8 points.OMP stayed within one point of a purpose-built modern selector.Source: §8 · p. 13
  • Roughly half the per-frame spatial budget could be removed with little observed change, within a post-hoc ±3-point interval on pooled LongVideoBench long videos; reinvesting those tokens in additional keyframes returned two to three points.Source: §8 · p. 13
  • The recommended order is to select relevant frames, compress them, then use the savings to see more of the video; benefits depend on video length, benchmark, and the model doing the reading.Source: §8 · p. 14

9 Ethical Considerations

The study uses existing public benchmarks and frozen models, collects no new human-subject data, and respects benchmark video licenses.

9 Ethical Considerations

  • The experiments use public video-QA benchmarks and frozen open or API models, collect no new subjects, personal data, videos, or annotations, and do not redistribute benchmark videos.Question stems, but not answer options, enter the selector; benchmark videos remain governed by their original licenses.Source: §9 · p. 14

A FOCUS⋆schedule replay

The FOCUS⋆ replay tests its temporal clip schedule under matched dense LongCLIP scoring, not the native FOCUS pipeline’s efficiency or accuracy. In native FOCUS, clip-arm pulls purchase ITM evidence (Zhu et al., 2025b).

A FOCUS⋆schedule replay

  • FOCUS⋆ replays the temporal clip schedule with already-computed LongCLIP scores, so it tests the schedule under a matched scorer rather than the full FOCUS pipeline.The original FOCUS treats scoring as a budgeted online process in which clip-arm pulls purchase ITM evidence (Zhu et al., 2025b); this replay draws no conclusion about native FOCUS efficiency or accuracy.Source: A FOCUS⋆schedule replay · p. 15
  • The replay uses 16 s clips, three coarse pulls per clip, α=0.25, a half-candidate-pool pull budget, and approximately k/4 surviving clips.Unobserved rewards are interpolated from the nearest observed frame, and the random seed is fixed per item; the row’s star marks this boundary from native FOCUS’s ITM protocol.Source: A FOCUS⋆schedule replay · p. 15

B Question categories do not define a stable selector regime

LongVideoBench’s category tags do not support a stable question-type frame-selection regime: OMP’s category interaction with uniform sampling was not significant in either long-duration bin, so these results do not justify a question-type router.

B Question categories do not define a stable selector regime

C Selector variants · D Exploratory failure audit

Selector variants changed less than replacing uniform sampling or reallocating pixels into frames, while the exploratory audit surfaced competing OMP failure patterns without establishing how prevalent they are. The selector sweep was internally paired in a secondary compute environment, not compared with Tables 2–5.

C Selector variants

  • No tested DPP variant significantly exceeds OMP in either long bin, and query-blind DPP (β=0) collapses toward uniform sampling.MMR underperforms, while query-weighted DPP, residual floors, and partial orthogonalization sit in a band around OMP.Source: C Selector variants · p. 16
  • The sweep arms are internally paired in a secondary compute environment and are never compared against Tables 2–5.Source: C Selector variants · p. 16
  • Within this scorer geometry and budget range, changing relevance–diversity rules produces less change than replacing uniform sampling or reallocating pixels into frames.The nulls do not establish that all diversity objectives are equivalent; the pattern is consistent with query-directed signal being exhausted after roughly five picks.Source: C Selector variants · p. 16

D Exploratory failure audit

  • 41 cases across 16 categories in the 600 s bin and 52 cases across 17 categories in the 3600 s bin were inspected for OMP failures at k=8.Two independent model-assisted visual passes reviewed selected timestamps, rendered frames, questions, gold answers, and model predictions; tags included missed evidence, temporal offset, answerer error, and visually unanswerable questions.Source: D Exploratory failure audit · p. 16
  • Visually distinctive but irrelevant content consuming several of eight picks was the most frequent tag in both failure-audit passes.Examples included unrelated compilation chapters, separate storylines, title and subscribe cards, and dark transitions; the sample cannot establish that this pattern is dominant, OMP-specific, or prevalent.Source: D Exploratory failure audit · p. 16
  • Sequence-of-scenes and cross-scene tracking questions also failed when dense top-k sampling covered one relevant moment but omitted another required scene.The audit motivates relevance-constrained diversity as a design direction, but cannot estimate population failure rates or establish that adding frames benefits any question category.Source: D Exploratory failure audit · p. 16

Related papers

Loading 2609.03820v1…