HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering

Dan Ben-Ami; Gabriele Serussi; Kobi Cohen; Chaim Baskin

Source-linked AI summary

HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering

Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, Chaim Baskin

arXiv:2603.18558v2Mar 19, 2026cs.CVcs.AI

TL;DR

Long-form VideoQA needs relevant evidence selected within finite MLLM context windows, while global query-frame scoring misses multimodal bindings and temporal structure. HiMu uses a training-free hierarchical logic tree with lightweight visual and audio experts and fuzzy-logic composition. It achieves state-of-the-art controlled-selector accuracy across three benchmarks and improves uniform sampling across seven MLLMs without iterative MLLM selection calls.

Problem
Finite MLLM context windows require long videos to be represented by a small frame budget, but global similarity selectors miss multimodal bindings and temporal ordering in compositional queries.
Method
HiMu makes one text-only LLM call to build a hierarchical logic tree, routes atomic predicates to visual or audio experts, and composes their timeline signals into a per-frame satisfaction curve.
Results
HiMu achieves state-of-the-art accuracy among controlled frame selectors across Video-MME, LongVideoBench, and HERBench-Lite, while improving uniform sampling for seven diverse MLLMs.
Takeaways & Limitations
Structured query representation can recover relevant context within controlled budgets, making HiMu a plug-and-play selector without MLLM calls during selection.
Takeaways & Limitations
HiMu depends on faithful tree parsing and expert coverage, especially ASR coverage for multilingual or noisy speech, while downstream MLLMs still must fuse evidence across selected frames.

Abstract

from arXiv · show

Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows. Within the controlled frame-budget regime that governs practical deployment, prior selectors score frames against a single global query embedding; as a result, compositional multimodal questions that involve temporal ordering or cross-modal cues such as ``what happens on screen right after the narrator mentions the reaction?'' are flattened into a representation that loses sub-event ordering and modality bindings. We introduce \textbf{HiMu}, a training-free framework for compositional multimodal frame selection. A single text-only LLM call decomposes the query into a hierarchical logic tree whose leaves are atomic predicates, each routed to a lightweight expert spanning vision (CLIP, open-vocabulary detection, OCR) and audio (speech recognition and non-speech sound matching). Expert signals are normalized, smoothed to align across modalities, and composed bottom-up through fuzzy-logic operators that enforce temporal sequencing and adjacency, yielding a continuous per-frame satisfaction curve. Under the standard 16-frame budget on Video-MME, LongVideoBench, and HERBench-Lite, HiMu achieves state-of-the-art accuracy among frame selection methods and improves over uniform sampling across seven diverse MLLMs as a drop-in module, matching the accuracy of uniform sampling at $4\times$ its frame budget, without retraining and without multiple iterative MLLM calls during selection.

1 Introduction

Long-form VideoQA must select a small set of relevant frames because MLLMs have finite context windows, but existing selectors flatten multimodal, temporally structured queries. HiMu addresses this gap with hierarchical decomposition, modality-specific experts, and fuzzy-logic composition, achieving strong benchmark results without MLLM calls during selection.

Motivation: Finite MLLM context windows make processing long videos at native frame rates infeasible, so systems operate under a fixed frame budget.Frame selection is a bottleneck because correct answers require the relevant visual evidence.
Motivation: Global similarity selectors collapse compositional queries into one score, preventing joint reasoning over audio, vision, and sub-event ordering.The motivating example requires narration, the beaker’s visual state, and their temporal order.
Motivation: The missing capability is a modality-complete, compositional selector that evaluates evidence across the timeline within controlled-budget costs.Complex queries may involve objects, actions, text, narration, and non-speech sounds.
HiMu: HiMu uses one text-only LLM call to decompose queries into hierarchical logic trees, routes leaves to visual and audio experts, and composes signals into per-frame satisfaction curves.Experts include CLIP, OVD, OCR, ASR, and CLAP; fuzzy-logic operators combine localized evidence bottom-up.
Results: Across Video-MME, LongVideoBench, and HERBench-Lite, HiMu achieves state-of-the-art accuracy among controlled frame selection methods and improves uniform sampling across seven MLLMs.The method is presented as a plug-and-play selector without model-specific tuning.

2 Related Work

Related work spans efficient similarity-based selectors, structured visual search, and agentic systems that repeatedly invoke multimodal models. HiMu remains in the controlled-budget regime while composing visual, speech, and non-speech evidence in one non-iterative pass.

Controlled selection: Controlled frame selectors identify informative frames within a fixed budget without invoking the downstream MLLM during selection.This defines the compute regime that distinguishes controlled selectors from agentic VideoQA systems.
Similarity-based selection: Similarity-based methods score frames against frozen vision-language embeddings, using diversity, recursive allocation, or determinantal point processes.BOLT, AKS, and MDP3 represent distinct ways to refine global similarity-based selection.
Structured selection: Structured selectors add explicit relations such as spatial co-occurrence, temporal proximity, attribute dependency, and causal order, often through iterative visual search.T* and VSLS exemplify detector-based or predefined-relation approaches.
Agentic systems: Agentic and multi-call systems trade additional computation for repeated MLLM or VLM reasoning inside the selection loop.They address answer-relevant evidence discovery but operate outside the controlled-selector compute regime.
HiMu: HiMu addresses the gap by composing visual, speech, and non-speech audio evidence through a hierarchical temporal-logic tree in one non-iterative selection pass.This contrasts query flattening, fixed-relation visual search, and multi-call systems.

3 Method

HiMu selects frames by decomposing questions into modality-specific logic trees, processing expert signals, and composing them with temporal fuzzy logic into a satisfaction curve used for frame selection.

3.1 Neuro-Symbolic Query Decomposition: HiMu parses each question into a hierarchical logic tree whose leaves route atomic predicates to CLIP, OVD, OCR, ASR, or CLAP experts.Internal operators represent co-occurrence, alternatives, chronological sequence, and tight right-after transitions.
3.2 Multimodal Expert Signals Extraction and Processing: Expert signals cover visual concepts, objects, on-screen text, spoken content, and environmental sounds, with cached features reused whenever possible.Only OVD is query-conditioned; unused experts are skipped entirely.
3.2 Multimodal Expert Signals Extraction and Processing: Signals are normalized robustly, smoothed with modality-specific Gaussian kernels, and bandwidth-matched so cross-modal peaks overlap during composition.Joint normalization preserves relative magnitudes among leaves sharing an expert, while wider kernels accommodate the coarser timing of ASR and CLAP.
3.3 Fuzzy Logic Composition: Bottom-up fuzzy-logic composition produces a per-frame satisfaction curve that combines co-occurrence, alternatives, temporal ordering, and tight temporal proximity.SEQ uses past and future evidence to enforce chronology, while RIGHTAFTER uses exponential decay and scores both cause and effect sides.
3.3 Fuzzy Logic Composition: PASS selects top frames from separated local peaks and their high-scoring neighbors, reducing concentration on one segment while retaining short-term context.The method uses minimum inter-peak distance and a local window around each peak.

4 Experiments

Experiments evaluate HiMu as a controlled frame selector across benchmarks, downstream MLLMs, compositional components, frame budgets, and latency. HiMu achieves the strongest controlled accuracy while providing broad plug-and-play gains at a modest selector cost.

Main Results: HiMu achieves the best controlled-selection scores on Video-MME, LongVideoBenchval, and HERBench-Lite.With Qwen3-VL-8B and K=16, scores are 73.2%, 64.2%, and 43.2%, improving over the strongest baseline by +3.2, +5.5, and +1.0pp, respectively.
Generalization: HiMu improves over uniform sampling across seven diverse MLLMs, with the largest gains on longer Video-MME videos.Video-MME Overall gains range from +4.4 to +7.1pp, while LongVideoBenchval gains range from +2.9 to +13.8pp.
Component Analysis: −5.5pp results from replacing the logic tree with Flat Fusion, exceeding the drop from removing any individual expert.Flattening the tree while retaining the root operator costs −1.4pp, isolating hierarchy as a major contributor to performance.
Efficiency Analysis: 4.6 s first-query and 1.9 s cached per-query latency position HiMu between lightweight retrieval selectors and heavier structured selectors.The comparison uses a 10-minute video at 1 FPS on 8× NVIDIA RTX 6000 Pro GPUs and excludes the final QA call.
Frame Budget: HiMu consistently improves Overall at different frame budgets and is especially strong on Medium and Long videos.The frame-budget evaluation uses Video-MME with Qwen3-VL-8B.

5 Discussion

HiMu demonstrates that compositional multimodal selection can operate within a controlled frame budget without MLLM calls during selection. Its structured representation treats audio as first-class evidence, while its main limitations concern selector cost, expert coverage, parsing fidelity, and downstream evidence fusion.

Discussion: HiMu performs compositional multimodal frame selection within a controlled budget without an MLLM call during selection.It uses a hierarchical logic tree with lightweight modality-specific experts, including compositionally typed audio evidence.
Discussion: Structured query representations can recover relevant context more directly than larger uniform budgets that provide redundant coverage.This is presented as a broader implication of controlled-budget selection.
Limitations: HiMu’s quality depends on faithful tree parsing and coverage of the underlying experts, especially ASR for multilingual or noisy speech.Expert extraction is slower than global-embedding retrieval, and downstream MLLMs must still fuse evidence across selected frames.

Supplementary Material

The supplementary material specifies how HiMu’s text-only parser constructs benchmark-adapted logic trees and routes atomic predicates to visual and audio experts. Worked examples illustrate typed operators, multimodal grounding, and valid JSON tree output.

Prompt Design: The parser converts each question into a hierarchical logic tree using benchmark-specific modality adaptations.The common prompt is shared across benchmarks, while expert descriptions and rules are enabled or disabled according to available audio modalities.
Logic Operators: AND, OR, SEQ, and RIGHT_AFTER encode co-occurrence, alternatives, explicit temporal order, and immediate succession.SEQ is used only when order is explicitly stated, while RIGHT_AFTER takes exactly two children representing cause and effect.
Prompt Design: The prompt encourages multimodal leaves, ASR keyword overlap, factored multiple-choice structure, and decomposition of rich descriptions.These rules aim to retain complementary evidence across expert types.
Output Format: The parser output is constrained to JSON with recursive operator nodes and expert-query fields on leaf nodes.Available experts are restricted to each benchmark’s active expert set.
Expert Routing: Physical objects route to OVD, visual actions and states to CLIP, on-screen text to OCR, spoken content to ASR, and environmental sounds to CLAP.Actions involving objects are represented with AND(OVD:object, CLIP:action), while proper names use OCR plus ASR.
Worked Example: A worked doorbell example grounds the cue in CLAP, the door-opening event in CLIP, and answer alternatives in an OVD-based OR branch.The RIGHT_AFTER root captures the local temporal relation between the ring and subsequent door opening.

B Implementation Details

The supplementary implementation details organize hyperparameters by pipeline stage and link them to the corresponding equations and main-paper sections.

Implementation Details: Table 7 groups all hyperparameter values by pipeline stage and references their associated equations and main-paper sections.

C PASS Algorithm

PASS selects temporally diverse, locally contextualized frames from HiMu’s satisfaction curve instead of concentrating the budget at one peak. Its ablations show stable accuracy across moderate score-processing changes, while over-smoothing visual signals is most harmful.

PASS selection procedure: Naive top-K concentrates selected frames around one satisfaction-curve peak, whereas PASS detects multiple peaks and provides diverse temporal coverage.Figure 5 contrasts single-peak concentration with multi-peak selection and neighbor spreading.
PASS selection procedure: PASS detects multiple separated peaks, adds high-scoring neighbors around each, and greedily fills the remaining frame budget.The three phases enforce temporal diversity, capture short-term motion context, and complete the selection.
PASS selection procedure: 73.23% versus 72.37% for vanilla top-K on full Video-MME with Qwen3-VL-8B and K=16, a +0.86 pp gain.PASS spreads frames across distinct temporal events while preserving fine-grained context around each key moment.
Sensitivity analysis: Hyperparameter perturbations remain within at most 0.71 pp of baseline, indicating limited sensitivity to smoothing, temporal decay, and sigmoid sharpness.The sensitivity experiments change one component at a time from the default configuration.
Sensitivity analysis: Visual over-smoothing with σ=2 is most harmful at −0.71 pp, while temporal decay varies by at most −0.08 pp across κ settings.Speech smoothing removal has 0.00 pp effect, and sigmoid alternatives remain within 0.56 pp of default.

D.2 Expert Backbone Ablation

HiMu’s accuracy is largely preserved when expert backbones are replaced, supporting backbone flexibility. The tree-parser comparison likewise reports a narrow accuracy range across diverse parser models.

Expert backbone swaps: All expert-backbone substitutions stay within 0.6 pp of default accuracy, with Grounding DINO improving by +0.52 pp and music-speech CLAP by +0.07 pp.These swaps indicate that practitioners can substitute backbones based on deployment constraints with minimal reported accuracy degradation.
LLM tree parser comparison: All four LLM tree parsers fall within a 0.78 pp accuracy range when Qwen3-VL-8B remains the downstream answerer.The parsers span open-source and proprietary models of different scales on the full Video-MME test set.
LLM tree parser comparison: The weakest parser, LLaVA-OV-1.5-8B, reaches 72.52%, only 0.71 pp below the default, while Gemini-2.5-Flash is marginally best at +0.07 pp.The reported spread supports robustness to the choice of tree parser under the tested setup.

D.4 Component Ablation Protocol

The component ablation protocol isolates evidence, modality availability, and structural choices by holding the relevant pipeline elements fixed. Structural substitutions alter operators or hierarchy while preserving the underlying expert evidence in most conditions.

Protocol: The full-tree reference scores 73.23% overall on full Video-MME with Qwen3-VL-8B and K=16 selected frames.The main-paper ablations use this configuration as the reference setting.
Expert ablation: Expert leave-one-out neutralizes every leaf of the ablated expert while fixing decomposition, topology, remaining outputs, answerer, and frame budget.Replacement uses parent-operator neutral constants: 1 under AND, SEQ, and RIGHTAFTER, and 0 under OR.
Visual-only ablation: HiMu-Visual reparses each question with only CLIP, OVD, and OCR exposed, measuring deployment when speech and non-speech audio experts are unavailable.The answerer and frame-selection setting otherwise define the adaptive visual-only configuration.
Structural ablations: Operator substitutions preserve the tree topology and leaves while changing only the fusion function after rescaling per-frame signals to [0.5, 1.0].The tested replacements respectively loosen conjunction, tighten disjunction, remove ordering, or remove RIGHTAFTER proximity coupling.
Structural ablations: NO_NESTING alone changes topology by wrapping depth-first leaves under the original root operator, making it milder than Flat Fusion.Flat Fusion additionally forces a generic SIMPLE_OR combination.

E Selector Latency Breakdown

HiMu separates cacheable video-level evidence extraction from question-specific selection. Under the reported 8-GPU protocol, caching reduces the first-query selector cost to a 1.88 s per-query critical path dominated by parsing and detection.

Measurement setup: A 10-minute video sampled at 1 FPS produces 600 candidate frames with K=16 on 8× RTX 6000 Pro GPUs; OCR uses 300 half-rate frames.The measurement covers selector-only forward-pass compute with weights loaded and prepared inputs already placed on devices.
Preprocessing: Cacheable preprocessing extracts CLIP, CLAP, OCR, and Whisper evidence once per video before question-specific composition.CLIP processes 600 frames, CLAP 300 two-second windows, OCR 300 sampled frames, and Whisper the full 600 s audio track.
Per-query processing: The per-query selector critical path is 1.88 s, while the first-query cost is 4.63 s including 2.74 s of cacheable preprocessing.Per-query stages include the text parser, conditional detection, lightweight matching, and CPU composition with PASS.
Interpretation: After caching, the query-specific path is dominated by the text-only tree parser followed by query-conditioned detection, matching, and composition.Most video evidence is reused across questions, while video-level extraction is parallelized across the 8 GPUs.

F Interpretability of Frame Selection

HiMu makes frame selection auditable by exposing per-predicate activations behind each deterministic satisfaction score. Its heatmaps connect these activations to temporal frame evidence and support diagnosis of perception, composition, and query-decomposition failures.

Auditable Selection: Each frame receives an explicit attribution vector whose entries record the activation strength of each expert-predicate pair.The vector accompanies the deterministic fuzzy-logic satisfaction score T(t), making the basis of selection inspectable.
Auditable Selection: Unlike similarity-based selectors’ single opaque scalar, HiMu identifies which query component and modality supported each selected frame.Other structured selectors expose narrower logs, while multi-call agents provide reasoning traces without an explicit scored ranking.
Leaf-Level Visualization: Figure 6 heatmaps place the K=16 selected frames in temporal columns and named logic-tree leaves in rows, with darker cells indicating stronger predicate satisfaction.The corresponding thumbnails allow visual verification of heatmap activations against video content.
Leaf-Level Visualization: Shared-context leaves for “black dog” and “in water” remain strongly active across frames, while answer-option leaves show distinct temporal activation patterns.For example, CLIP: “submerging” scores 1.00 on every frame, whereas CLIP: “searching” and OVD: “fish” are dark only in the earlier cluster.
Diagnostic Value: The heatmap can localize errors to an expert, a temporal operator, or the query decomposition, turning frame selection into a diagnostic tool.Examples include OCR missing on-screen text, incorrect SEQ ordering, and overly generic predicates.