Source-linked AI summary
GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs
Mohamed Eltahir, Lama Ayash, Ali Habibullah, Tanveer Hussain, Naeemullah Khan
TL;DR
Long-video VLMs face costly dense attention, while existing selectors lack QA-answer-space, cross-frame scoring and closed-form per-question frame budgeting. GridProbe runs row and column posterior probes over a K × K frame grid, then uses the importance map’s shape to set an adaptive per-question frame budget. Single-model GridProbe trades 1.63 pp Avg Acc for 3.36× TFLOPs reduction on V2 and is Pareto-dominant on LVB; 2B→8B Pareto-dominates 2B-monolithic on both (+3.56 pp Avg Acc at 0.83× compute on V2).
Problem
Method
GridProbe runs row and column posterior probes over a K × K frame grid, then uses the importance map’s shape to set an adaptive per-question frame budget.
Source: §1 · p. 2, §3.3 · p. 5Results
Single-model GridProbe trades 1.63 pp Avg Acc for 3.36× TFLOPs reduction on V2 and is Pareto-dominant on LVB; 2B→8B Pareto-dominates 2B-monolithic on both (+3.56 pp Avg Acc at 0.83× compute on V2).
Source: §4.2 · p. 8Takeaways & Limitations
Per-question Meff tracks intrinsic question difficulty without observing answers, supporting test-time adaptive compute.
Source: §6 · p. 10Compute savings are more modest with long prompts or K<10 grids, and cross-model use requires both selector and QA models in host memory simultaneously.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Long-video understanding in VLMs is bottlenecked by a single monolithic forward pass over thousands of frames at quadratic attention cost. A common mitigation is to first select a small subset of informative frames before the forward pass; common for training-free selectors via auxiliary encoder-space similarities. Such signals are capped by contrastive pretraining, which usually fails on reasoning-heavy queries (negation, cross-frame counting, holistic summarization). We propose GridProbe, an efficient training-free posterior-probing inference paradigm that scores evidence in answer space using a frozen VLM's own reasoning and then selects question-relevant frames adaptively, resulting in sub-quadratic attention cost with little to no accuracy loss. We arrange frames on a $K{\times}K$ grid and run lightweight row R and column C probes, where each probe reads its peak posterior as a query-conditioned confidence. The outer product of R and C yields an interpretable importance map whose skewness and kurtosis drive Shape-Adaptive Selection, a closed-form rule that reliably replaces the fixed frame budget $M$ with a per-question $M_{\mathrm{eff}}$. We show empirically that $M_{\mathrm{eff}}$ tracks intrinsic question difficulty without ever seeing the answer, a sign of test-time adaptive compute. On Video-MME-v2, GridProbe matches the monolithic baseline within $1.6$ pp Avg Acc at $3.36\times$ TFLOPs reduction, while on LongVideoBench it Pareto-dominates the baseline ($+0.9$ pp at $0.35\times$ compute). Because the selector and QA models can be decoupled, pairing a small 2B selector with a stronger 4B or 8B QA is strictly Pareto-dominant over the 2B monolithic baseline (up to $+4.0$ pp at $0.52\times$ compute, on average), with no retraining. Finally, the interpretability of the importance maps opens future avenues for behavioral diagnostics, grounding, and frame-selection distillation.
1 Introduction
GridProbe replaces one-shot long-video processing and encoder-space frame selection with frozen-VLM posterior probes that score evidence in answer space. Its importance map also supports per-question frame budgets, adapting compute to question difficulty.
Introduction
- Long-video VLMs face a compute–fidelity trade-off: dense attention is costly, while encoder-space selectors can miss reasoning-heavy queries such as negation, counting, and summarization.The introduction notes that Qwen3-VL-2B uses ≈240 visual tokens per frame at 2048 frames versus ≈2960 at 64 frames, and that attention cost grows quadratically with sequence length.Source: §1 · p. 2
- GridProbe uses lightweight row and column probes with a frozen VLM; their peak answer posteriors form a question-conditioned frame-importance map for selecting evidence.By default, the same frozen VLM acts as selector and answerer; the selector and QA roles can also be decoupled.Source: §1 · p. 2
- The answer-space importance map makes the VLM’s frame-level evidence gathering interpretable and grounds selection in the backbone’s reasoning capacity rather than contrastive embeddings.The introduction says this supports queries involving negation, cross-frame counting, and compositional reasoning; the current formulation reads a peak posterior over a finite answer space.Source: §1 · p. 2
- The map’s shape drives a closed-form per-question M_eff, replacing fixed frame budgets that can waste compute on localized questions or discard context needed for holistic ones.The selection rule uses skewness and kurtosis, and the introduction reports that importance-distribution shape strongly correlates with question difficulty.Source: §1 · pp. 2–3
- Positive-skew sparse-peak maps and negative-skew redundant-high-importance maps can yield the same frame-selection answer under GridProbe’s Redundancy Principle.The introduction also identifies Pareto-dominant cross-model composition without retraining, Pareto-efficient single-model operation, and adaptive compute that mirrors intrinsic difficulty as empirical findings.Source: §1 · pp. 2–3
2 Related Work
Prior work reduces long-video inference cost through encoder-space selection, multimodal frame scoring, or agentic compute allocation, but leaves gaps in reasoning-aware scoring and per-question budget adaptation.
Long-video VLMs and inference cost
Encoder-space frame selection
- Encoder-space selectors use CLIP [8] and SigLIP [9] similarities, but reasoning-heavy queries can defeat signals disconnected from the QA model’s reasoning.FOCUS [4] adapts exploration, MDP3 [2] optimizes subsets, and learned Frame-Voyager [3] and HFS [5] add scoring heads or fine-tuning complexity.Source: §2 · p. 3, Figure 3 · p. 4
Multimodal scoring and static budgets
- FRAG [10] scores frames with a multimodal model but remains frame-wise, while fixed M can overspend on localized queries and underserve holistic ones.Figure 3 contrasts encoder-space similarity from separate vision and text encoders with answer-space confidence from the QA VLM’s answer posterior.Source: §2 · p. 3, Figure 3 · p. 4
Test-time compute and agentic inference
Related work synthesis
- Prior approaches lack a fixed-schedule, training-free method that scores in QA answer space, captures cross-frame reasoning, and sizes each question’s budget in closed form.The paper identifies this as the gap GridProbe addresses in the following section.Source: §2 · p. 4
3 Methodology: GridProbe
GridProbe uses a frozen VLM’s answer posterior to estimate query-relevant evidence, factorizes frame scoring into row and column probes, and adapts the number of selected frames to the importance map’s shape.
3.1 Setup and Notation
- The probe confidence is the frozen VLM’s peak answer posterior, used as a query-conditioned relevance proxy: high confidence suggests evidence, while a flat posterior suggests insufficient discrimination.Confidence indicates relevance, not correctness; this answer-space signal differs from independent encoder-space similarity scoring.Source: §3.1 · p. 4
3.2 Grid Formulation and Importance Map
- With K² sampled frames arranged as a K×K grid, row and column probes combine local and periodic confidence into cell weights in 2K passes, rather than the K² per-frame passes of prior scoring [10].A cell receives high importance only when both its row and column are confident; the grid factorization produces a question-conditioned map.Source: §3.2 · p. 5
3.3 Adaptive Selection Size via Distribution-Shape Statistics
- Skewness and excess kurtosis of the importance map drive an adaptive frame budget: peaked or redundant evidence needs fewer frames, while near-uniform evidence calls for broad coverage.A perfectly uniform map falls back to K² frames, while a one-hot map tends toward one; the method combines the two shape moments into σ(M).Source: §3.3 · pp. 5–6
3.4 Two-Stage Inference Pipeline
- GridProbe probes K row and K column subsets, builds the importance map and adaptive budget, then runs one full-resolution pass on the top-Meff frames to produce the answer.The focused pass returns the answer with the highest posterior probability given the selected frames and query.Source: §3.4 · p. 6
- On illustrative Video-MME-v2 examples, Meff ranges from 140 frames for a holistic query to 5 for a specific one; GridProbe answers the specific query correctly with 5 frames while the 144-frame baseline fails.The paper also reports that MDP3 with its default budget of 8 misses both the holistic and specific examples.Source: Figure 5 · p. 7
3.5 Complexity
- For N=K² frames, probing costs O(N^1.5) attention plus O(Meff²) for the focused pass, sub-quadratic when Meff is much smaller than N; uniform maps revert to the monolithic cost.In the reported setup, probes use reduced 224×224 resolution, making the probe passes cheaper than full-resolution inference.Source: §3.5 · p. 7
4 Experiments
GridProbe uses a frozen VLM’s posterior probes to select frames and adapt compute per question. Experiments report competitive benchmark accuracy at reduced compute and show that compute allocation varies with question difficulty.
4.1 Experimental Setup
- Experiments use frozen Qwen3-VL-Instruct 2B, 4B, and 8B backbones on Video-MME-v2 without subtitles and LongVideoBench with subtitles, reporting accuracy and per-question TFLOPs.Unless stated otherwise, K=12 creates a 144-frame candidate pool, with frames sampled uniformly from the video timeline.Source: §4.1 · p. 7
4.2 Main Results: Video-MME-v2 and LongVideoBench
- On Qwen3-VL-2B, GridProbe(M=AUTO) trades 1.63 pp Avg Acc on V2 for 3.36× TFLOPs reduction, and gains +0.9 pp on LVB at 0.35× compute.The V2 result is a trade-off against the same-size monolithic baseline; the LVB point is Pareto-dominant.Source: Table 1 · p. 8, §4.2 · p. 8
- Pairing a 2B selector with an 8B QA Pareto-dominates the 2B monolithic baseline: +3.56 pp Avg Acc at 0.83× compute on V2 and +3.30 pp at 0.85× on LVB.The 2B→4B pipeline gains +4.0 pp on LVB at 0.52× compute; against matched-compute uniform frame selection, importance ranking adds +0.50 pp on V2 and +1.20 pp on LVB.Source: Table 1 · p. 8, §4.2 · p. 8
- At fixed M=8, GridProbe and MDP3 match on V2 but diverge on LVB, where GridProbe wins; switching from fixed M=8 to M=AUTO adds +0.90 Non-Lin pp and +4.1 Long-bin pp at near-matched compute.MDP3 lands 4.9 pp below the no-selection baseline on the LVB 3600-second bin, indicating encoder-space scoring can be actively harmful there.Source: §4.2 · pp. 8–9, Table 2 · p. 8, Table 3 · p. 8
4.3 Adaptive Compute Mirrors Intrinsic Question Difficulty
- GridProbe’s Meff is lowest at both extremes of importance-map skewness and near-full for near-uniform maps, while baseline accuracy mirrors this pattern at about 21% versus 28–32%.Compute CV is 0.78 for GridProbe versus 0.018 for the fixed-input baseline, at 0.30× the per-question average compute; allocation tracks intrinsic difficulty without observing answers.Source: §4.3 · p. 9
5 Ablation Study
The ablations test selector size, focused-pass frame ordering, and image collation, generally showing trade-offs between compute and accuracy. Unless stated otherwise, they use Qwen3-VL-2B at K=12 on Video-MME-v2 with M=AUTO.
Selector size
- With QA fixed at 2B, scaling the selector from 2B to 8B decreases Avg Acc and increases TFLOPs despite mean Meff remaining similar (52.9 → 56.2 → 58.5).The results suggest a stronger selector may choose frames suited to its own reasoning capacity rather than the smaller QA model; cross-model amortization is one-directional, pairing a small probe with a stronger answerer.Source: §5 · p. 9
Frame ordering
- Descending-importance ordering of the same focused-pass frames costs −1.25 pp Avg Acc versus temporal ordering.Temporal order preserves the positional encoding the VLM was trained to read.Source: §5 · p. 9
Image collation
- 0.29× the standard two-stage compute with a −1.16 pp Avg Acc cost makes image collation a credible option under extreme compute constraints.The focused pass tiles the top-Meff frames into one 2048×2048 image, sacrificing temporal positional encoding and per-frame pixel budget.Source: §5 · p. 10
6 Conclusion and Limitations
GridProbe uses row and column probes over a K×K frame grid to form a question-conditioned importance map and set a per-question budget. It reports efficient single-model results and cross-model Pareto dominance, while noting compute, memory, and answer-format limitations.
Conclusion
- GridProbe’s shape statistic sets a per-question frame budget from a question-conditioned importance map, and its cross-model variant Pareto-dominates the 2B-monolithic baseline on both benchmarks without retraining.The single-model method is Pareto-efficient on Video-MME-v2 and Pareto-dominant on LongVideoBench; Meff tracks intrinsic question difficulty without observing the answer.Source: §6 · p. 10
Limitations
- TFLOPs savings are smaller when prompts are large or K<10, and the cross-model pipeline trades compute for host memory by loading both models simultaneously.LongVideoBench subtitles add ∼700–1,000 tokens that every probe re-processes; at K<10, the 2K probe passes become a non-trivial share of total cost.Source: §6 · p. 10
- Future refinements include adapting γ0 to video length or pool density and applying the shape statistic to other importance signals; open-ended QA remains future work.Probe confidence is defined for the finite answer space of multiple-choice benchmarks, so generalization to open-ended QA is non-trivial.Source: §6 · p. 10
A Implementation Details
Implementation uses zero-shot Qwen3-VL backbones in a two-stage probe-and-focus pipeline. The section specifies its defaults, benchmark protocols, and evaluation hardware.
Models
- Experiments use zero-shot Qwen3-VL-Instruct backbones at 2B, 4B, and 8B parameters, with answers selected by letter-token scoring.There is no fine-tuning, chain-of-thought prompting, or tool use; LongVideoBench uses its five-way answer set.Source: A Implementation Details · p. 12
Hyperparameters
- Defaults are K=12, γ0=0.25, and a small variance threshold for setting σ=0; probes run at 224×224, while focused frames use native resolution and adaptive token allocation..Source: A Implementation Details · p. 12
Pipeline
- The pipeline probes 144 uniformly sampled frames with 24 row/column passes, ranks frames using an outer-product importance map, and focuses one native-resolution pass on the question-specific budget.Probe passes use 12-frame tiled images; the budget is adjusted using the map's shape statistics.Source: A Implementation Details · p. 12
Benchmarks
- Video-MME-v2 visual-only evaluation covers 3,200 questions across 800 videos and four duration bins, reporting Non-Lin and Avg Acc.LongVideoBench validation includes subtitles in both stages' prompts and reports Long and Overall accuracy.Source: A Implementation Details · p. 12
Hardware
- Evaluation ran on one node with 8× NVIDIA A100 GPUs, partitioning samples across GPUs by interleaved sample ID to balance duration bins.Per-shard JSON outputs are merged; source code is slated for public release upon publication.Source: A Implementation Details · p. 12
B Detailed Breakdown Tables
This section provides the per-bin breakdown referenced in Table 1.
B Detailed Breakdown Tables
- The section presents a per-bin breakdown referenced from Table 1.Source: B Detailed Breakdown Tables · p. 12
B.1 Video-MME-v2: Per-Level and per-Group-Type
On Video-MME-v2, GridProbe’s adaptive selection performs best on transition-focused temporal understanding and loses most on distributed multi-point aggregation. Its per-level and group-type breakdown distinguishes where selective evidence helps from where broad coverage is needed.
B.1 Video-MME-v2: Per-Level and per-Group-Type
- Video-MME-v2 reports Non-Lin scores across three cognitive levels and two group types; Consistency contains 519 of 800 groups and Coherence 281 of 800.The levels are Information Aggregation, Temporal Understanding, and Complex Reasoning; the group types are capability-consistency and reasoning-coherence groups.Source: §B.1 · p. 12
- Against the same-model 2B baseline, GridProbe(M=AUTO) gains ∆=+0.37 Non-Lin on Level 2 temporal understanding and loses ∆=−1.92 Non-Lin on Level 1 multi-point aggregation.The passage attributes the gain to evidence concentrated at a transition-bearing event and the loss to evidence distributed across many timestamps, where no small subset suffices.Source: §B.1 · p. 12
- The σ-driven adaptive M degrades gracefully toward K2 on near-uniform maps, avoiding overcommitment when evidence is dispersed but offering no benefit when it is already fully covered.Source: §B.1 · p. 12
B.2 LongVideoBench: Full Per-Duration Breakdown
GridProbe’s duration-bin results show that adaptive selection helps when uniform sampling undersamples long videos, but can hurt when short videos are already fully covered.
B.2 LongVideoBench: Full Per-Duration Breakdown
- At 600-sec and 3600-sec, question-conditioned selection picks relevant moments where uniform K2 undersamples the timeline; at 15-sec and 60-sec, 144 uniform frames already saturate it.The passage attributes the short-bin losses to selection being unnecessary once the timeline is fully covered.Source: §B.2 · p. 12
- The duration-bin pattern parallels V2’s cognitive-level results: GridProbe earns its keep when evidence is sparse and degrades gracefully when evidence is already fully covered.Source: §B.2 · p. 12
- GridProbe(M=AUTO) gains +2.2 pp at 600-sec and +1.6 pp at 3600-sec, but loses −3.1 pp at both 15-sec and 60-sec bins.The gains and losses are reported against the corresponding no-selection baseline.Source: §B.2 · p. 12, Table 8 · p. 13