Source-linked AI summary
GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs
Mohamed Eltahir, Lama Ayash, Ali Habibullah, Tanveer Hussain, Naeemullah Khan
TL;DR
Long-video VLMs need to reduce frame processing without losing question-relevant evidence, but existing selectors do not score in the QA model’s answer space or adapt the frame budget. GridProbe uses training-free posterior probes and shape-adaptive selection to achieve Pareto-efficient or Pareto-dominant performance across benchmarks, including +0.9 pp at 0.35× compute on LongVideoBench.
Problem
Long-video VLMs face costly dense frame processing, while existing selectors lack QA-model answer-space scoring, cross-frame reasoning, and closed-form per-question budget sizing.
Method
GridProbe probes row and column frame subsets with a frozen VLM, combines posterior confidences into an importance map, and uses its shape to set M_eff.
Results
GridProbe is Pareto-efficient on Video-MME-v2 and Pareto-dominant on LongVideoBench, achieving +0.9 pp at 0.35× compute on LongVideoBench.
Takeaways & Limitations
Per-question M_eff tracks intrinsic question difficulty without observing the answer, and decoupling selector and QA models yields Pareto improvements without retraining.
Takeaways & Limitations
Compute savings are more modest with large prompts or small grids, and cross-model use shifts cost toward host memory because both models must be loaded.
Abstract
from arXiv · showhide
Long-video understanding in VLMs is bottlenecked by a single monolithic forward pass over thousands of frames at quadratic attention cost. A common mitigation is to first select a small subset of informative frames before the forward pass; common for training-free selectors via auxiliary encoder-space similarities. Such signals are capped by contrastive pretraining, which usually fails on reasoning-heavy queries (negation, cross-frame counting, holistic summarization). We propose GridProbe, an efficient training-free posterior-probing inference paradigm that scores evidence in answer space using a frozen VLM's own reasoning and then selects question-relevant frames adaptively, resulting in sub-quadratic attention cost with little to no accuracy loss. We arrange frames on a $K{\times}K$ grid and run lightweight row R and column C probes, where each probe reads its peak posterior as a query-conditioned confidence. The outer product of R and C yields an interpretable importance map whose skewness and kurtosis drive Shape-Adaptive Selection, a closed-form rule that reliably replaces the fixed frame budget $M$ with a per-question $M_{\mathrm{eff}}$. We show empirically that $M_{\mathrm{eff}}$ tracks intrinsic question difficulty without ever seeing the answer, a sign of test-time adaptive compute. On Video-MME-v2, GridProbe matches the monolithic baseline within $1.6$ pp Avg Acc at $3.36\times$ TFLOPs reduction, while on LongVideoBench it Pareto-dominates the baseline ($+0.9$ pp at $0.35\times$ compute). Because the selector and QA models can be decoupled, pairing a small 2B selector with a stronger 4B or 8B QA is strictly Pareto-dominant over the 2B monolithic baseline (up to $+4.0$ pp at $0.52\times$ compute, on average), with no retraining. Finally, the interpretability of the importance maps opens future avenues for behavioral diagnostics, grounding, and frame-selection distillation.
1 Introduction
GridProbe replaces encoder-space, fixed-budget frame selection with training-free posterior probing in answer space. It uses question-conditioned importance-map shape to select evidence and adapt compute per question, while exposing the VLM’s evidence gathering.
- Motivation: Encoder-space selectors score frames through vision–text embedding similarity, limiting selection when queries require negation, counting, or compositional reasoning.GridProbe instead asks the VLM which subsets support confident answers in answer space.
- Method: GridProbe runs lightweight row and column probes on a K × K frame grid, then forms a question-conditioned importance map from their peak-posterior confidences.The outer product of row and column confidences produces the map, replacing the standard one-shot forward pass.
- Design consequences: The posterior-probing signal inherits the VLM’s reasoning capacity, scales with backbone capability without retraining, and yields mechanically interpretable evidence maps.This supports native handling of negation, cross-frame counting, and compositional queries.
- Adaptive compute: A closed-form statistic replaces the fixed frame budget M with a per-question M_eff determined by the importance distribution’s shape.The map shape correlates with question difficulty, allowing localized questions to use less computation and holistic questions to retain context.
- Adaptive compute: Positive-skew sparse-peak and negative-skew redundant-high-importance maps can share the same selection answer, motivating shape-aware selection beyond skew direction alone.This is the Redundancy Principle underlying the adaptive budget rule.
2 Related Work
Related work reduces long-video inference cost through encoder-space frame selection, multimodal frame scoring, or agentic routing, but each leaves a complementary gap. The missing direction is a fixed-schedule, training-free mechanism that adapts question-specific evidence selection and compute to the QA model’s reasoning.
- Long-video VLMs and the cost of monolithic inference: Long-video VLMs process thousands of frames with a single monolithic forward pass whose attention cost is quadratic in input length.Extended context windows and adaptive per-frame visual-token budgets do not make dense attention over dense sampling affordable.
- Encoder-space frame selection: Encoder-space selectors reduce input volume using vision-language similarities, but their signals remain disconnected from the QA model’s reasoning.Methods include CLIP- and SigLIP-based scoring, adaptive exploration, list-wise subset optimization, and learned auxiliary scoring heads.
- Multimodal frame scoring and the static-budget assumption: Multimodal frame scoring moves selection into model space, yet frame-wise top-M selection lacks temporal context and reasoning about evidence sufficiency.Prior methods also fix the selection size M a priori, which can waste compute on localized queries.
- Test-time compute and agentic video inference: Agentic video inference routes compute per question through multi-step orchestration, complementing but not replacing a fixed-schedule, training-free mechanism.VideoAtlas uses a hierarchical video grid explored by a Master-Worker agent loop and achieves logarithmic compute growth with video duration.
3 Methodology: GridProbe
GridProbe uses a frozen VLM’s posterior confidence to build a question-conditioned frame-importance map from complementary local and periodic probes. Shape-adaptive selection then converts the map’s distribution into a per-question frame budget for a focused final pass.
- Probe confidence: Posterior peak confidence measures how confidently the frozen VLM commits to an answer and serves as a proxy for whether a frame subset contains query-relevant evidence.High confidence indicates relevance, whereas a flat posterior suggests insufficient evidence to discriminate among candidates.
- Grid probing: The K×K grid combines K row probes for local temporal coverage with K column probes for strided periodic coverage, requiring 2K passes over K-frame subsets.Rows expose localized event evidence, while columns sample distributed or recurring evidence across the full timeline.
- Importance map: Multiplying row and column confidences produces a cell-level importance map that emphasizes frames supported by both local and global evidence.A cell receives moderate weight when only one marginal is confident and is downweighted when neither is confident.
- Shape-Adaptive Selection: Skewness and excess kurtosis summarize the importance-map shape, allowing sparse-peak and redundancy regimes to receive fewer frames while near-uniform maps receive broader coverage.The method sets σ=0 for a perfectly uniform map, where standardized moments are undefined.
- Inference pipeline: The focused pass selects the top Meff entries of the importance map and runs the frozen VLM once at full resolution to produce the final answer.The method first records probe confidences and computes Meff, then evaluates arg maxy pθ(y | S⋆, q) on the selected frames.
4 Experiments
Experiments on Video-MME-v2 and LongVideoBench show that GridProbe substantially reduces compute while preserving or improving accuracy, especially when pairing a 2B selector with stronger QA models. Results also support adaptive, question-dependent frame selection and a cheaper collated-image variant.
- Experimental setup: Experiments use frozen Qwen3-VL-Instruct backbones at 2B, 4B, and 8B on Video-MME-v2 and LongVideoBench, with K=12 producing a 144-frame candidate pool.Video-MME-v2 is evaluated without subtitles, while LongVideoBench includes subtitles.
- Same-model results: 1.63 pp lower Avg Acc on Video-MME-v2 accompanies a 3.36× TFLOPs reduction for 2B GridProbe, while LongVideoBench gains +0.9 pp at 0.35× compute.These are same-model comparisons against monolithic baselines at K=12.
- Cross-model results: +3.56 pp Avg Acc at 0.83× compute on Video-MME-v2 and +3.30 pp at 0.85× on LongVideoBench are achieved by GP-2B→8B versus the 2B-monolithic baseline.GP-2B→4B yields +4.0 pp at 0.52× on LongVideoBench, widening to +4.5 pp on its 3600-sec bin.
- Single-image variant: 20.11 accuracy and 66.6 compute are reported for the collated one √Meff×√Meff image, versus 21.27 and 232.2 for the standard Meff-frame video two-stage variant.The no-selection baseline reports 23.10 accuracy and 819.6 compute.
- Adaptive selection: GridProbe’s adaptive selection varies with importance-map shape: both positive- and negative-skew extremes route to small Meff, while near-uniform maps receive near-full coverage.The result empirically supports selecting different frame counts for different question-content distributions.
5 Ablation Study
The ablations test selector size, focused-pass frame ordering, and image collation under a common Video-MME-v2 setup. Results show that larger selectors can hurt accuracy, temporal ordering preserves performance, and tiled-image collation offers substantial compute savings with a modest accuracy cost.
- Ablation setup: The ablations use Qwen3-VL-2B at K=12 on Video-MME-v2 with M=AUTO, varying selector size, frame ordering, and image collation.All three ablations target components of the adaptive two-stage inference pipeline.
- Selector size: Selector scaling from 2B to 8B decreases Avg Acc while increasing TFLOPs despite roughly stable average M_eff values of 52.9 →56.2 →58.5.The degradation is attributed to stronger selectors identifying frames aligned with their own reasoning capacity rather than the fixed QA model.
- Frame ordering: −1.25 pp Avg Acc results when the same top-M_eff frames are passed in descending-importance rather than temporal order.Temporal ordering preserves the positional encoding the VLM was trained to read.
- Image collation: 0.29× the standard two-stage’s compute is achieved by compositing top-M_eff frames into one tiled image, at a −1.16 pp Avg Acc cost.This variant sacrifices temporal positional encoding and per-frame pixel budget, making it suitable when extreme compute reduction is the priority.
6 Conclusion and Limitations
GridProbe is a training-free posterior-probing method that uses row-column probes and a shape statistic to set question-specific frame budgets. It enables Pareto-efficient or Pareto-dominant operation, while its compute benefits weaken with large prompts or small grids and its cross-model variant increases host-memory demands.
- Method: GridProbe uses row and column probes over a K×K frame grid to construct a question-conditioned importance map and set the per-question budget M_eff.The budget is determined by a closed-form shape statistic.
- Results: GridProbe provides Pareto-efficient single-model operation on Video-MME-v2 and Pareto-dominant operation on LongVideoBench.The cross-model pipeline also Pareto-dominates the 2B-monolithic baseline on both benchmarks without retraining.
- Limitations: TFLOPs reductions are more modest with large prompts, such as LVB’s ∼700–1,000 subtitle tokens, because every probe re-processes them.The reduction is most pronounced when the focused pass dominates inference.
- Limitations: For small grids with K<10, the 2K probe passes become a non-trivial fraction of total cost.This reduces the practical compute advantage of posterior probing.
- Limitations: The cross-model pipeline shifts resource demands from compute toward host memory because the selector and QA models are loaded simultaneously.This is a systems-level cost of pairing separate selector and QA models.
A Implementation Details
GridProbe uses zero-shot Qwen3-VL-Instruct models with a two-stage, posterior-based frame-selection pipeline. The implementation fixes the frame grid, probe resolution, adaptive budget rule, benchmark protocols, and distributed evaluation setup.
- Models: 2B, 4B, and 8B Qwen3-VL-Instruct backbones perform zero-shot inference without fine-tuning, chain-of-thought prompting, or tool use.Answers are selected by arg max_y pθ(y | ·) over answer-letter tokens.
- Hyperparameters: K=12 yields a 144-frame pool, with γ0=0.25 controlling the per-question Meff rule and a variance threshold handling near-uniform maps.Probes use 224×224 resolution, while focused inference uses native resolution and adaptive per-frame token allocation.
- Pipeline: 144 uniformly sampled frames undergo 24 probe passes, each tiling 12 frames into a 2048×2048 image for row and column confidence scoring.The importance map is M[r, c] = c_r · c_c, and each pass contributes its top-1 answer-letter confidence.
- Benchmarks: Video-MME-v2 evaluates 3,200 visual-only questions across 800 videos, while LongVideoBench validation reports Long and Overall accuracy with subtitles included in both stages.Video-MME-v2 spans 15s, 60s, 600s, and 3600s duration bins and reports Non-Lin and Avg Acc.
- Hardware: 8× NVIDIA A100 GPUs run the experiments, with evaluation sharded by sample ID and outputs merged from per-shard JSON files.Interleaved partitioning balances duration-bin distributions across shards.
B Detailed Breakdown Tables
This section provides the per-bin breakdown referenced from Table 1.
- The section presents the per-bin breakdown referenced from Table 1.
B.1 Video-MME-v2: Per-Level and per-Group-Type
On Video-MME-v2, GridProbe(M=AUTO) varies by cognitive level: it improves most on temporal understanding but struggles with distributed multi-point aggregation. The adaptive budget degrades gracefully toward K2 when evidence maps are near-uniform.
- Dataset breakdown: Video-MME-v2 groups questions into L1 Information Aggregation, L2 Temporal Understanding, and L3 Complex Reasoning across Consistency and Coherence types.Consistency contains 519 of 800 groups, while Coherence contains 281 of 800 groups; Table 7 reports Non-Lin scores by level and group type.
- Per-level results: +0.37 Non-Lin: GridProbe(M=AUTO) beats the same-model 2B baseline on Level 2 temporal understanding.The passage attributes this gain to evidence concentrated at a transition-bearing event.
- Per-level results: −1.92 Non-Lin: GridProbe(M=AUTO) performs worst against the same-model 2B baseline on Level 1 multi-point aggregation.Evidence is distributed across many timestamps, so no small subset suffices; on near-uniform maps, σ-driven adaptive M degrades gracefully toward K2.
B.2 LongVideoBench: Full Per-Duration Breakdown
GridProbe’s duration breakdown shows that question-conditioned selection helps most on long videos, where relevant evidence is sparse, while uniform sampling is sufficient for short videos. It wins at 600 and 3600 seconds but loses at 15 and 60 seconds, with the same graceful pattern across cognitive levels.
- Duration-dependent performance: GridProbe(M=AUTO) wins on the 600-sec (+2.2 pp) and 3600-sec (+1.6 pp) duration bins.Question-conditioned selection picks relevant moments where uniform sampling undersamples the timeline.
- Duration-dependent performance: GridProbe(M=AUTO) loses on the 15-sec and 60-sec bins (−3.1 pp each).Uniformly sampled frames already saturate short timelines, making selection unnecessary.
- Cross-breakdown pattern: The same pattern appears across V2 cognitive levels: GridProbe helps when evidence is sparse and degrades gracefully when evidence is already fully covered.The duration-dependent behavior is consistent with the cognitive-level breakdown referenced in §B.1.