PaperlayerView exact revision on arXiv

Source-linked AI summary

FORTE: Adaptive Scoring and Exact Keyframe Selection for Long-Video Question Answering

Haifeng Huang, Biyin Xu, Chunsheng Xin, Yang Li

arXiv:2610.00573v1cs.CV

TL;DR

Uniform candidate sampling can miss question-relevant frames, while denser scoring raises costs and selection must also preserve temporal coverage. FORTE adaptively scores frames and optimizes a final relevance–coverage subset; across four benchmarks, it achieves the highest observed mean accuracy among compared methods under tested budgets.

  • Problem

    Uniform candidate sampling can miss relevant frames, while denser scoring raises costs; selection must also balance relevance with temporal coverage.

    Source: §1 · p. 2

  • Method

    FORTE adaptively scores frames using Gaussian-process relevance predictions and coverage, then exactly optimizes the final relevance–coverage subset of scored candidates.

    Source: §1 · p. 2

  • Results

    Across four benchmarks, FORTE has the highest observed mean accuracy among compared methods under tested budgets, with consistent effectiveness across relevance scorers and downstream MLLMs.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Accuracy gains diminish as the relevance-scoring budget increases, highlighting the importance of allocating evaluations effectively rather than simply scoring more frames.

    Source: §1 · p. 2

Abstract

from arXiv · show

Query-aware keyframe selection enables multimodal large language models (MLLMs) to process long videos using only a small set of question-relevant frames. Existing score-based methods, however, typically search within a fixed, uniformly sampled candidate pool, preventing evidence outside this pool from ever being selected. Given a limited relevance-scoring budget, the key challenge is to allocate evaluations adaptively to promising frames while continuing to explore underrepresented temporal regions. We introduce FORTE, a training-free framework that addresses this challenge through two stages: adaptive relevance scoring and global keyframe optimization. Starting from sparse, uniformly distributed observations, our efficient Gaussian-process relevance predictor estimates relevance for unscored frames, exploiting temporal locality and the approximately banded kernel structure to reduce the core computation from cubic to linear time in the number of frames for fixed bandwidth. The scoring stage then selects which frames to score next by balancing predicted relevance with temporal coverage, prioritizing promising regions while also exploring less-represented parts of the video. The optimization stage selects the final keyframes by maximizing an objective that jointly captures measured relevance and temporal coverage. We derive an exact algorithm that leverages the logarithmic coverage structure to identify the optimal subset of the scored candidate pool in time linear in the pool size, for a fixed final-frame budget. Experiments on four long-video question-answering benchmarks show that FORTE achieves the highest observed mean accuracy among the compared selectors under every tested scoring budget. Further evaluations demonstrate its consistent effectiveness across different relevance scorers and downstream MLLMs.

1 INTRODUCTION

Long-video MLLMs need to preserve question-relevant evidence within limited frame and scoring budgets, but uniform sampling can miss relevant frames and denser evaluation costs more. FORTE adaptively scores frames and optimizes the final set for relevance and coverage, achieving the highest observed mean accuracy among compared selectors across tested budgets.

1 INTRODUCTION

  • Across four benchmarks, FORTE achieves the highest observed mean accuracy among compared selectors across tested relevance-scoring and final-frame budgets.Figure 1(b) reports four-benchmark mean QA accuracy with Qwen3-VL-8B and 64 input frames.Source: Figure 1 · p. 2, §1 · p. 2
  • Uniform sampling can miss question-relevant frames, while denser relevance evaluation raises computation and memory costs, motivating a limited scoring budget.Existing score-based methods commonly evaluate uniformly sampled frames.Source: §1 · pp. 1–2
  • FORTE coordinates adaptive relevance scoring and global keyframe optimization under a unified objective that balances relevance with temporal coverage.The scoring stage uses predictions and coverage gains to choose evaluations; final selection optimizes measured relevance and coverage.Source: §1 · p. 2
  • Its Gaussian-process predictor exploits temporal locality and an approximately banded kernel structure to reduce core computation from cubic to linear time.Predictions for unscored frames are updated as new scores are observed.Source: §1 · p. 2
  • A monotone upper-envelope algorithm exactly optimizes the final-frame objective in O(KN) time for scored-pool size N and final-frame budget K.The algorithm exploits the objective’s logarithmic coverage structure.Source: §1 · p. 2

2 RELATED WORK

Long-video systems address extended temporal context through longer-context modeling, memory, and visual compression, while keyframe selection determines which frames reach the model. Prior selection approaches include reasoning-heavy evidence search and lightweight scoring, but do not jointly adapt frame-level scoring to relevance and temporal coverage.

2.1 VIDEO LLMS FOR LONG-VIDEO UNDERSTANDING

  • Long-video systems extend context, aggregate historical observations, or compress visual representations; keyframe selection instead decides which frames are supplied to the model.LongVILA uses long-context training and sequence parallelism; MovieChat and MA-LMM use memory, while LongVU and Video-XL compress visual representations.Source: §2.1 · p. 3

2.2 KEYFRAME SELECTION METHODS

  • Keyframe selection includes LLM-guided evidence search and score-based methods, but reasoning workflows can add overhead and score-based methods typically score uniformly sampled frames.Prior methods do not effectively combine frame-level relevance estimates and temporal coverage to adaptively choose which frames to score; FORTE updates GP predictions between batches and balances predicted relevance against marginal coverage.Source: §2.2 · p. 3

3 METHOD

FORTE coordinates adaptive relevance scoring and global keyframe optimization around a shared relevance–coverage objective, while keeping the scorer and downstream MLLM frozen. It predicts unscored-frame relevance, uses coverage to guide both stages, and exactly optimizes the final subset of scored candidates.

3.1 PROBLEM FORMULATION AND OVERVIEW OF FORTE

  • FORTE starts with sparse, uniformly spaced observations, then predicts relevance and balances it with temporal coverage to allocate the remaining scoring budget.The scoring budget B and final-frame budget K are specified independently.Source: §3.1 · pp. 3–4, §3.2 · p. 4

3.2 ADAPTIVE RELEVANCE SCORING

  • The GP predictor estimates relevance for unscored frames from observed scores and timestamps, exploiting temporal locality and an approximately banded kernel for linear-time, linear-memory fitting at fixed neighborhood width.Predictions in regions without nearby observations revert to the observed mean, while a diagonal adjustment stabilizes near-singular kernels.Source: §3.2 · pp. 4–5
  • Coverage is based on temporal gaps: equal boundary-inclusive gaps maximize coverage, while marginal gains are nonnegative and diminish as more frames are added.The gain depends only on the gap containing the candidate, enabling local updates as predicted relevance and coverage guide each scoring choice.Source: §3.2 · pp. 5–6

3.3 GLOBAL KEYFRAME OPTIMIZATION

  • Global optimization balances measured relevance with coverage of the retained subset, and an exact dynamic program returns a global maximizer in O(KN) time for a fixed scored pool.The algorithm uses the logarithmic gap reward to maintain an upper envelope of predecessor lines; it uses O(KN) storage including backtracking indices.Source: §3.3 · p. 6

4 EXPERIMENTS

Across four long-video QA benchmarks, FORTE leads the compared selectors at every tested scoring budget, with only small average-accuracy gains from doubling the scoring allowance from ρ = 0.5 to 1.0. The experiments also examine its two-stage design, frame-budget and model generalization, and computational cost.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses four complementary benchmarks, reports multiple-choice accuracy and an unweighted four-benchmark mean, and tests four downstream models, three relevance scorers, and scoring allowances ρ ∈ {0.1, 0.5, 1.0} with K = 64.Qwen3-VL-8B-Instruct and BLIP are the default downstream model and relevance scorer.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • FORTE achieves the highest observed accuracy on all four benchmarks at every tested scoring budget; increasing ρ from 0.5 to 1.0 brings only small average-accuracy gains despite roughly doubling scorer evaluations.The authors suggest ρ = 0.5 offers a practical balance between accuracy and scoring cost.Source: §4.2 · p. 8

4.3 ABLATION AND GENERALIZATION STUDIES

  • With global optimization fixed, full adaptive scoring outperforms uniform scoring and either single-term variant; global optimization also improves on Top-K for both uniform and adaptive scored pools.The scoring variants combine predicted relevance and coverage, while global optimization maximizes their joint relevance–coverage objective.Source: §4.3 · p. 8, Table 2 · p. 8
  • FORTE has the highest observed mean accuracy at every tested final-frame budget, and with 16 frames matches the reported mean accuracy of Uniform with 64.All six compared methods improve as the final input-frame budget increases.Source: §4.3 · pp. 8–9, Table 3 · p. 8
  • With Qwen3-VL scales from 2B to 32B, accuracy increases with model size and FORTE remains ahead of AKS, WFS-SB, KFC, and FOCUS at every tested scale.Figure 3 reports four-benchmark mean accuracy.Source: Figure 3 · p. 9, §4.3 · p. 9

4.4 COMPUTATIONAL COST

  • At the default scoring budget, decoding and scorer inference account for approximately 72% of FORTE runtime, while adaptive scoring and optimization take 0.88 seconds (3.4%) of its 25.56-second latency.Total latency is close to that of Top-K, AKS, WFS-SB, and KFC (24.87–25.52 seconds); all scored methods reach the same selection-stage GPU-memory peak of 2.52 GiB.Source: Figure 4 · p. 9, §4.4 · p. 9

4.3 ABLATION AND GENERALIZATION STUDIES

  • FORTE outperforms Top-K, AKS, WFS-SB, and KFC in mean accuracy with BLIP, CLIP, and BLIP-2 relevance scorers.Selector hyperparameters are kept fixed when BLIP is replaced.Source: Table 4 · p. 8, §4.3 · p. 9
  • Across four downstream MLLMs, the method achieves the highest observed mean accuracy among compared selectors on each model, although the strongest baseline varies across models.The named evaluated models are LLaVA-OneVision-7B, LLaVA-Video-7B, and InternVL3.5-8B.Source: §4.3 · p. 9

5 CONCLUSION

FORTE uses separate budgets to choose which frames to score and which to retain, balancing relevance with temporal coverage. Its exact algorithm finds the globally optimal subset of the scored pool in O(KN) time, and its coverage objective favors balanced spacing with diminishing gains from nearby frames.

5 CONCLUSION

  • FORTE makes adaptive scoring and final frame retention decisions under separate budgets, balancing relevance with temporal coverage in both stages.The conclusion reports highest observed mean accuracy among compared methods across four benchmarks and consistent generalization across evaluated scorers and downstream MLLMs.Source: §5 · p. 9

A COVERAGE OBJECTIVE AND SELECTION GUARANTEES

  • The exact algorithm maximizes the final relevance–coverage objective over the scored pool in O(KN) time by exploiting its logarithmic gap structure.The coverage scale τ > 0 remains fixed as the selected set changes.Source: A COVERAGE OBJECTIVE AND SELECTION… · p. 14
  • The coverage objective favors balanced temporal spacing at a fixed frame count, with decreasing gain from adding frames near existing selections.These properties are the basis for the exact selection algorithm.Source: A COVERAGE OBJECTIVE AND SELECTION… · p. 14

A.1 TEMPORAL GAPS AND UNIFORM-SPACING OPTIMALITY

The appendix formalizes logarithmic temporal coverage and shows how its ordered-gap structure supports exact optimization; it also describes adaptive scoring updates.

A.1 TEMPORAL GAPS AND UNIFORM-SPACING OPTIMALITY

  • For coverage alone, equal temporal gaps maximize the objective, but relevance and a scored-candidate restriction can make the optimal subset nonuniform.The equal-spacing result applies when any timestamp can be selected.Source: §A.1 · p. 14

A.2 MARGINAL COVERAGE AND DIMINISHING RETURNS

  • Adding a frame cannot decrease coverage, and the coverage function has diminishing returns, making the final relevance–coverage objective monotone submodular under the stated conditions.Its gap decomposition also permits exact optimization rather than generic approximate greedy maximization.Source: §A.2 · p. 15

A.3 EXACT ALGORITHM

  • The exact optimizer uses a dynamic-programming recurrence and a monotone upper envelope to find the optimal subset in O(KN) time.It uses O(N) working memory plus O(KN) storage for backtracking indices.Source: §A.3 · pp. 15–16

B DETAILS OF ADAPTIVE RELEVANCE SCORING

  • Adaptive scoring starts from sparse uniform observations, predicts unscored-frame relevance, and repeatedly selects and scores frames using both predicted relevance and temporal coverage.New measured scores update the predictor until the scoring budget is exhausted.Source: B DETAILS OF ADAPTIVE RELEVANCE SCORING · p. 17

B.1 GAUSSIAN-PROCESS RELEVANCE PREDICTOR

The predictor approximates GP relevance locally using temporally neighboring observations, then combines predictions with coverage to acquire frames in batches and update the model between rounds.

B.1 GAUSSIAN-PROCESS RELEVANCE PREDICTOR

  • The GP predictor uses measured scores and timestamps, retaining a band around the kernel matrix diagonal and restricting each prediction to nearby observations.Predictions approach the observed-score average where no scored frames are nearby.Source: §B.1 · pp. 17–18
  • For fixed bandwidth, GP preparation, fitting, and prediction take O(|U|) time per round and use O(|U|) memory.Across R rounds, the stated GP cost is O(R|U|).Source: §B.1 · p. 18
  • Frame acquisition ranks unscored candidates by predicted relevance plus marginal coverage, updating affected candidates in the selected frame’s temporal gap.The stated acquisition and heap-maintenance cost is O(R|U| + BG0 log |U|).Source: §B.1 · p. 19

B.2 BATCH CONSTRUCTION AND OBSERVATION UPDATES

  • Each scoring round selects a batch using fixed GP predictions while updating coverage after each choice, then scores the batch in parallel and updates observations for the next round.Batching avoids a GP refit after every individual evaluation.Source: §B.2 · p. 20

C.1 FORTE IMPLEMENTATION

The experiments use fixed FORTE defaults and a shared video QA pipeline, then compare matched-budget selectors, scoring policies, and exact versus greedy optimization.

C.1 FORTE IMPLEMENTATION

  • FORTE’s defaults include an RBF length scale of 10 seconds, four scoring rounds, approximately half-budget uniform initialization, βacq = 3.5, and βsel = 1.The authors report using shared heuristic defaults without development-set tuning and robustness to moderate parameter changes.Source: §C.1 · p. 21

C.2 EVALUATION PROTOCOL

  • The evaluation covers four benchmarks using video frames without audio or separately supplied subtitles, and reports question-weighted accuracy with an unweighted four-benchmark mean.The evaluated sets include LongVideoBench, Video-MME, LVBench, and MLVU.Source: §C.2 · pp. 21–22

C.3 BASELINE IMPLEMENTATIONS

  • Except for FOCUS, score-based baselines evaluate a uniformly sampled candidate pool; TOP-K then retains its highest-scoring frames.Budget-matched fixed-grid baselines need not score the same frames as FORTE.Source: §C.3 · p. 23

D ADDITIONAL EXPERIMENTAL RESULTS

  • Unless otherwise stated, experiments use Qwen3-VL-8B-Instruct, BLIP, ρ = 0.5, K = 64, and exact global keyframe optimization.Benchmark accuracies are question-weighted, while Avg. weights the four benchmarks equally.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 23

D.1 MATCHED-BUDGET COMPARISON WITH FOCUS

  • 65.8% versus 62.4% mean accuracy: FORTE exceeds FOCUS on all four benchmarks when both receive matched scoring budgets.The comparison uses BLIP, Qwen3-VL-8B-Instruct, and K = 64.Source: §D.1 · p. 24, Table 6 · p. 24

D.2 CONTRIBUTIONS OF THE TWO STAGES

  • With global optimization, combining predicted relevance and coverage reaches 66.1% mean accuracy, versus 65.2% for uniform scoring and 64.2% for relevance-only scoring.Coverage-only scoring matches uniform scoring at 65.2%.Source: §D.2 · p. 24, Table 7 · p. 25
  • Global optimization improves mean accuracy over Top-K from 64.6% to 65.2% on uniform pools and from 64.9% to 66.1% on adaptive pools.The reported larger gain on adaptive pools indicates complementarity between the two stages.Source: §D.2 · p. 24, Table 7 · p. 25

D.3 EFFECT OF THE FINAL OPTIMIZATION ALGORITHM

  • Exact optimization raises mean accuracy by 0.3 points on uniform pools and 0.4 points on adaptive pools, at an additional 3 ms per selection.Approximate greedy optimization is slightly higher on LVBench, so accuracy does not improve on every benchmark.Source: §D.3 · p. 25, Table 8 · p. 25

D.4 PAIRED UNCERTAINTY ANALYSIS

Paired video-cluster comparisons support a mean-accuracy advantage for FORTE over four baselines at the default setting. Sensitivity sweeps find broad high-performing parameter regions, with relatively little benefit from finer tuning in several cases.

D.4 PAIRED UNCERTAINTY ANALYSIS

  • Video-cluster bootstrap intervals quantify sampling uncertainty across videos, conditional on the evaluated models and settings, rather than variation across inference runs.The analysis uses 20,000 replicates, resampling whole videos while keeping questions from each sampled video together.Source: §D.4 · pp. 25–26
  • All four 95% confidence intervals for four-benchmark mean differences favor FORTE over Uniform, AKS, FOCUS, and WFS-SB at the default setting.FORTE also has higher observed accuracy in every benchmark-level comparison, with intervals excluding zero against Uniform on all four benchmarks.Source: §D.4 · p. 26, Table 9 · p. 26

D.5 PARAMETER SENSITIVITY

  • Accuracy rises to a broad high-performing region at βacq = 3–5; every positive scoring coverage weight beats relevance-only scoring.The default βacq = 3.5 lies within that region.Source: §D.5 · p. 26
  • Accuracy peaks at βsel = 0.5 but remains similar from 0.25 to 1.5, including the retained unit-weight default.The authors retain the default rather than the sweep’s best-performing value, so the main results do not use a weight selected to maximize evaluation accuracy.Source: §D.5 · p. 26
  • Accuracy varies by only 0.2 points across initialization fractions of 0.25–0.75, while all three beat spending the entire budget on uniform scoring.The 0.5 initialization fraction has the highest result.Source: §D.5 · p. 26
  • More frequent GP updates improve accuracy at a fixed scoring budget, but doubling rounds from four to eight adds only 0.1 points.Four rounds retain most of the observed benefit with fewer refits.Source: §D.5 · p. 26
  • At ρ = 0.5, all tested GP length scales lie within 0.2 points; at ρ = 0.1, accuracy rises through 10 seconds and dips slightly at 20 seconds.The shared 10-second default is highest or joint-highest at both budgets.Source: §D.5 · p. 26

D.6 NUMERICAL VALIDATION OF THE GP PREDICTOR

The banded GP predictor substantially reduces fitting-and-prediction time with a small observed accuracy difference from dense GP. Validation also reports successful scoring replay, close acquisition agreement, low selection overhead, unchanged peak GPU memory, and stronger observed results on longer videos.

D.6 NUMERICAL VALIDATION OF THE GP PREDICTOR

  • All 23,280 question–budget pairs reproduce their saved scored pools exactly; all 91,247 GP fits succeed, with failed attempts accounting for 0.097% of GP time.The successful half-bandwidth is at most 128 for 99.45% of fits, and diagonal adjustment never exceeds 10^-3.Source: §D.6 · pp. 27–28, Table 10 · p. 27, Table 11 · p. 28
  • Banded and dense GP choose the same next frame in 424 of 451 sets (94.0%), with a mean absolute prediction difference of 0.00706 on tested pools.The comparison uses identical observations and clips predictions to the normalized score range.Source: §D.6 · p. 28
  • Dense GP exceeds banded GP by 0.1 percentage point in four-benchmark mean accuracy, while banded GP cuts fitting-and-prediction time from 15.07 to 0.81 seconds per question.The measured speedup is approximately 18.6×; individual-benchmark absolute differences are at most 0.2 points.Source: §D.6 · p. 28, Table 12 · p. 29

D.7 COMPUTATIONAL COST

  • FORTE spends 0.83 seconds on adaptive scoring and 0.05 seconds on final optimization, together 3.4% of its 25.56-second total runtime.Decoding and scorer inference dominate runtime; FORTE’s latency is within 0.7 seconds of compared fixed-grid baselines.Source: §D.7 · p. 29, Table 13 · p. 30
  • All scored methods reach peak allocated GPU memory of 2.52 GiB during selection and 23.46 GiB during answering; FORTE increases neither peak.Uniform’s selection-stage memory is unreported.Source: §D.7 · p. 29, Table 13 · p. 30

D.8 DURATION BREAKDOWNS

  • FORTE leads overall on LongVideoBench and Video-MME, with its strongest duration-group benefits on long videos, though other baselines lead some groups.On LongVideoBench, FORTE leads the 1 h group and ties for best in the 15 s group; on Video-MME, it leads the short and long groups.Source: §D.8 · p. 30, Table 14 · p. 30

D.9 SCORING-BUDGET REGIMES

The scoring-budget regime determines whether the scored pool can supply all K final frames, and completion may still be needed when B ≥ K. FORTE’s observed advantage persists in B ≥ K, while B < K comparisons are more mixed and confounded by completion.

D.9 SCORING-BUDGET REGIMES

  • When B < K, scored-pool selectors supplement frames to reach K; when B ≥ K, selection can remain within the scored pool if enough usable candidates remain.Completion may still be required at B ≥ K when fewer than K usable candidates remain.Source: §D.9 · p. 30
  • Questions in the B < K regime fall from 52.5% at ρ = 0.1 to 16.2% at 0.5 and 8.0% at 1.0.At the default budget, only LongVideoBench and Video-MME contain such questions; these counts are not supplementation rates.Source: §D.9 · p. 30, Table 15 · p. 31
  • Pooled across questions at ρ = 0.5 and K = 64, 8.13% of FORTE’s input frames are supplemental and 91.87% are scored.Equal benchmark weighting gives a supplemented fraction of 8.64%.Source: §D.9 · p. 31, Table 16 · p. 31
  • Within B ≥ K, FORTE has the highest observed four-benchmark mean at both scoring budgets, but B < K results are more mixed.B < K inputs for budget-matched score-based methods include unscored supplemental frames, so accuracy reflects both scored-pool construction and completion; differing group membership prevents isolating either effect.Source: §D.9 · p. 31

D.10 FURTHER MECHANISM CONTROLS

Mechanism controls support the selected GP and coverage components, while predictor comparisons favor GP over smoother alternatives. In acquisition tests, temporal coverage yields larger accuracy gains than posterior uncertainty and the better observed accuracy–cost trade-off.

D.10 FURTHER MECHANISM CONTROLS

  • Replacing GP prediction with linear interpolation or nearest observed scores reduces accuracy on all four benchmarks; centering the GP raises mean accuracy from 65.7% to 66.1%.Centering improves every benchmark, supporting a nonzero baseline in regions without nearby observations.Source: §D.10 · p. 32, Table 18 · p. 33
  • Freezing coverage gains within each scoring batch lowers accuracy, supporting frame-by-frame coverage updates even when relevance predictions remain fixed.Both square-root gap reward and removal of boundary-gap rewards also reduce accuracy on all four benchmarks.Source: §D.10 · p. 32, Table 18 · p. 33

D.11 COMPARISON WITH SMOOTH RELEVANCE PREDICTORS

  • GP prediction leads every benchmark at both budgets, improving mean accuracy over RBF regression by 0.7 points at ρ = 0.5 and 1.1 points at ρ = 0.1.The GP adds 0.28–0.41 seconds over RBF regression while remaining below one second of fitting-and-prediction time per question.Source: §D.11 · p. 33, Table 19 · p. 33

D.12 POSTERIOR UNCERTAINTY VERSUS TEMPORAL COVERAGE

  • At both tested budgets, coverage yields larger accuracy gains than posterior uncertainty; adding uncertainty to coverage leaves mean accuracy unchanged at ρ = 0.1 and slightly lowers it at 0.5.Adding uncertainty to FORTE increases acquisition time from 0.65 to 1.00 seconds at ρ = 0.1 and from 0.83 to 1.33 seconds at ρ = 0.5, without improving mean accuracy.Source: §D.12 · p. 34, Table 20 · p. 34

E EXTENDED ANALYSIS OF RELATED WORK

Long-video modeling methods improve how video information is represented and retained, while keyframe selection determines which frames enter the visual input. FORTE couples adaptive candidate acquisition with exact optimization of the final frame set.

E.1 VIDEO LLMS FOR LONG-VIDEO UNDERSTANDING

  • LongVILA, memory-based systems, and compression methods improve video representation and retention, whereas keyframe selection chooses the frames supplied to the model.The section describes LongVILA’s context scaling, MovieChat and MA-LMM’s historical memory, and hierarchical representations or compression in LongVLM, LongVU, Video-XL, VideoChat-Flash, and FLoC.Source: §E.1 · p. 35

E.2 KEYFRAME SELECTION METHODS

  • Training-based selectors learn policies from supervision or feedback, while LLM-guided search uses semantic judgments but can require costly captioning, analysis, and reasoning calls.FORTE uses frozen models and a lightweight relevance scorer without calling the downstream answer model during selection.Source: §E.2 · p. 35

E.2.1 SCORE-BASED FRAME SELECTION

  • Score-based methods refine selection from relevance-scored candidates, but a fixed candidate pool cannot recover evidence omitted before scoring.Prior approaches also combine relevance with temporal coverage, relevance-focused clips, semantic boundaries, or visual diversity.Source: §E.2.1 · p. 35
  • FORTE couples adaptive scoring of candidate frames with exact global optimization balancing measured relevance and temporal coverage (Section 3.3).Its logarithmic gap objective admits an exact O(KN) algorithm.Source: §E.2.1 · p. 35

F QUALITATIVE ANALYSIS

These selected examples distinguish retaining useful evidence from acquiring frames absent from a uniformly scored pool. They also show that even when relevant evidence is retained, the downstream model may answer incorrectly.

F QUALITATIVE ANALYSIS

  • These are selected examples rather than an estimate of how frequently each evidence-acquisition or retention behavior occurs.The comparisons distinguish evidence in scored pools from evidence retained in final inputs.Source: F QUALITATIVE ANALYSIS · pp. 36, 40

Retaining a readable view

  • FORTE alone retains a readable keyboard view near the requested timestamp and answers correctly; proximity alone does not ensure the object is identifiable.Top-K and AKS keep an earlier frame showing only an instrument edge, while Uniform and FOCUS omit the shot.Source: F QUALITATIVE ANALYSIS · p. 36

Linking distant scenes

  • FORTE retains both the restroom reference and direct beach views more than nine minutes later, and answers “beach” where AKS and FOCUS answer “suspension bridge.”AKS and FOCUS retain beach-adjacent scenes, so the comparison is not simply about whether beach imagery appears.Source: F QUALITATIVE ANALYSIS · p. 36

Preserving brief evidence

  • FORTE and Uniform retain the courtroom follow-up and answer correctly, while Top-K, AKS, and FOCUS omit the interval despite some having scored it.FOCUS retains several similar reporter views instead.Source: F QUALITATIVE ANALYSIS · p. 36, Figure 6 · p. 37

Retaining a readable view

  • FORTE and Top-K retain the full-size photograph of the red toy and answer correctly, while AKS and FOCUS retain less readable views and answer incorrectly.Uniform also answers correctly without a frame in the narrow full-size-photo interval.Source: F QUALITATIVE ANALYSIS · p. 36, Figure 6 · p. 38

Acquiring evidence absent from the scored pool

  • FORTE alone acquires and retains the brief hand gesture and bag fastener detail missed by uniform scoring, answering both questions correctly.Adaptive scoring found the gesture at 102.37 seconds and the fastener at 454.20 seconds; all methods returned 64 frames without supplementation.Source: F QUALITATIVE ANALYSIS · p. 36, Figure 6 · p. 39

A shared failure despite retained evidence

  • All methods answer “white” despite retaining views of the woman’s black dress, so the error is not explained solely by missing views of the clothing.The saved responses do not establish whether subject confusion or another aspect of reasoning caused the error.Source: Figure 7 · p. 40, F QUALITATIVE ANALYSIS · p. 40
Loading 2610.00573v1…