PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

Haoyu Guo, Yuan Feng, Junlin Lv, Mingjun Xiao, S Kevin Zhou, Xike Xie

arXiv:2609.16722v1cs.AIcs.CLcs.CVcs.MM

TL;DR

Long-form video MLLMs face visual-token growth that strains context windows and computational budgets, while existing reduction methods trade selection quality against efficiency. VideoMM screens regions on a downscaled Macro Proxy and activates high-resolution Micro Tokens only when needed. It reports a 6.13× speedup and 7.0% accuracy gain over vanilla baselines, and over 2.73× faster inference than state-of-the-art methods with comparable accuracy on LongVideoBench.

  • Problem

    Long-video token reduction must balance selection precision and computational overhead, while large visual-token volumes constrain context windows and increase inference costs.

    Source: §1 · p. 1, (b) GLM-4.1V-9B · p. 1

  • Method

    VideoMM selects relevant regions using a downscaled Macro Proxy and activates high-resolution Micro Tokens only when macro-level inference is insufficient.

    Source: (b) GLM-4.1V-9B · p. 1, §3.3 · p. 4

  • Results

    6.13× average speedup and 7.0% accuracy gain over vanilla baselines, with over 2.73× faster inference than state-of-the-art methods at comparable accuracy on LongVideoBench.

    Source: (b) GLM-4.1V-9B · p. 1

  • Takeaways & Limitations

    VideoMM presents adaptive perceptual granularity as a direction beyond model-centric token reduction, using coarse-to-fine processing to balance efficiency and semantic depth.

    Source: (b) GLM-4.1V-9B · p. 2, §5 · p. 8

Abstract

from arXiv · show

Scaling Multimodal Large Language Models (MLLMs) to long-form video understanding is bottlenecked by the explosion of visual tokens, which saturates context windows and incurs prohibitive costs. Current solutions predominantly rely on auxiliary models for token reduction but face a fundamental dilemma: lightweight encoder-driven approaches often overlook critical semantic information, whereas heavyweight MLLM-driven reduction negates the efficiency gains. {In this work, we identify a more fundamental inefficiency underlying this dilemma: while fine-grained visual details are essential for detailed understanding, they are largely redundant for the preliminary task of selecting semantically relevant regions. } Motivated by this, we introduce \textbf{VideoMM}, which marks a paradigm shift from model-centric downsizing to adaptive perceptual granularity. Specifically, our framework {decouples selection from reasoning} by executing semantic filtering on a cost-effective \textit{Macro Proxy} (derived from downscaled frames), and projecting the selected regions onto high-fidelity \textit{Micro Tokens} for detailed understanding only when necessary. Extensive evaluations show that VideoMM significantly outperforms existing solutions. It achieves a 6.13$\times$ speedup and a 7.4\% accuracy gain over full-context baselines on LongVideoBench, and further accelerates inference by 2.73$\times$ over current leading methods, establishing a highly scalable paradigm for long-video understanding. Our code is available at: https://github.com/adfh917k/VideoMM.

1 Introduction

Long-form video understanding remains challenging despite MLLMs’ success on static images and short clips, as visual-token volume strains context windows and raises computational costs. Existing token-reduction approaches use auxiliary models but face a fundamental dilemma.

  • MLLMs have succeeded on static images and short clips, but extending them to long-form video understanding remains a critical challenge.Source: §1 · p. 1
  • Long-video analysis supports applications such as video QA, temporal event localization, and long-horizon action recognition, but faces scalability bottlenecks.Source: §1 · p. 1
  • A 10-minute 720p video sampled at 1 fps generates over 300k visual tokens in Qwen-3-VL, rapidly exhausting context windows and incurring prohibitive computational costs.Source: §1 · p. 1
  • Existing token-reduction approaches use auxiliary models before full inference, yet face a fundamental dilemma.Source: §1 · p. 1

(b) GLM-4.1V-9B

VideoMM targets the accuracy–efficiency dilemma in token selection by using different perceptual fidelities for region selection and detailed reasoning. On LongVideoBench, it reports faster inference with higher accuracy than the vanilla baseline.

(b) GLM-4.1V-9B

  • 6.13× inference speedup and 7.4% accuracy gain over the vanilla baseline, plus 2.73× speedup over FlexSelect without accuracy loss, are reported on LongVideoBench.Figure 1 presents these comparisons.Source: Figure 1 · p. 1
  • Existing token-reduction methods struggle to balance accuracy and efficiency because inference stages require different levels of perceptual fidelity.Fine-grained details matter for precise understanding but are largely unnecessary for initially identifying relevant regions.Source: (b) GLM-4.1V-9B · p. 1
  • VideoMM selects relevant regions with a lightweight Macro Proxy and activates high-resolution Micro Tokens only when fine-grained inference is necessary.The framework uses a consensus mechanism to govern adaptive computation.Source: (b) GLM-4.1V-9B · p. 1

2 Related Works

Prior token-reduction methods use either lightweight encoders that may miss video semantics or auxiliary MLLMs that add computational overhead. VideoMM shifts the focus from model scale to perceptual granularity, using a macro-level proxy for token reduction.

2 Related Works

  • Lightweight encoder-driven methods [3] [25] [38] use ViT features to prune or compress tokens, but may erroneously discard semantically critical information.VisionZip [38] uses attention and embedding similarity, while VidCom uses embedding diversity to set frame-wise compression ratios.Source: §2 · p. 2
  • MLLM-driven methods [7] [41] [42] improve token selection accuracy, but auxiliary-model overhead can negate the latency gains from reduction.FlexSelect [42] uses a reduced-size MLLM and cross-modal attention to select tokens.Source: §2 · p. 2
  • VideoMM shifts from model scale to perceptual granularity, reporting faster inference than lightweight encoder-driven methods while preserving MLLM-driven accuracy advantages.The approach constructs a macro-level proxy for fast token reduction.Source: §2 · p. 2

3 Methods

VideoMM uses a macro-to-micro pipeline: a downscaled proxy supports semantic region selection, while high-resolution details are recruited only when macro-view inference is insufficient. Grouped selection uses the proxy to identify informative regions under MLLM guidance.

3 Methods

  • A standard video MLLM encodes sampled frames into visual tokens, combines them with text-query tokens, and generates a response autoregressively.The described model consists of a ViT encoder and a large language model.Source: §3 · p. 2

3.1 Overall Architecture of VideoMM

  • VideoMM's two-stage architecture uses a macro proxy to isolate relevant temporal regions, then assesses macro-view sufficiency and activates micro tokens when needed.A consensus mechanism governs whether inference terminates with the global context or recruits high-resolution details.Source: §3.1 · p. 2, Figure 2 · p. 3

3.2 Stage I: Grouped Selection with Macro Proxy

  • Spatial downscaling reduces selection cost while preserving the spatial consistency of critical features and the global layout of salient regions.The macro proxy lets the selection stage avoid processing raw high-resolution frames.Source: §3.2 · p. 3
  • Grouped selection partitions proxy frames into groups so the MLLM can independently build a sparse, global understanding of each group.The grouped strategy is used to identify informative tokens within context constraints.Source: §3.2 · p. 3

1. Spatial Downscaling to Macro Proxy

VideoMM estimates visual-token relevance using text-to-visual attention weights, then filters redundancy hierarchically at frame and token levels. It uses the resulting macro token set for subsequent inference.

1. Spatial Downscaling to Macro Proxy

  • Cross-modal attention weights from text queries provide the visual-token relevance signals used for selection.The selected macro tokens are formed after relevance scoring.Source: §1 · pp. 3–4
  • Frame importance is the sum of its constituent token scores, and each group retains its top 0.5 fraction of frames.The retained frames form a refined subset for subsequent token selection.Source: §1 · p. 4
  • After frame filtering, the method selects B tokens uniformly from each frame group to construct the macro-level token set.The authors use a fixed selection of three intermediate layers; the frame fraction also reduces the cost of re-encoding micro tokens.Source: §1 · p. 4

1. Macro-Consensus Verification

VideoMM checks whether independent macro-group responses agree, returning a consensus early or recruiting high-resolution micro tokens when they diverge. Its efficiency analysis attributes reduced attention cost to grouping, downscaling, and conditional refinement.

3.3 Stage II: Adaptive Macro-Micro Inference

  • When independent macro groups and the jointly selected macro set agree, VideoMM returns their consensus response without micro-level processing.Divergent macro-view responses indicate ambiguity and trigger high-resolution refinement.Source: §3.3 · pp. 4–5
  • On divergence, VideoMM encodes only the α-fraction of frames selected in Stage I, maps selected macro tokens to corresponding micro representations, and uses them for the definitive response.The resulting micro-token set is the union of the selected spatial-region subsets.Source: §3.3 · p. 5

3.4 Efficiency Analysis

  • The expected refinement cost is conditional on consensus failure: (1−β)(ρN)^2, where ρ is the ratio of selected high-resolution tokens.Macro processing uses the downscaled proxy divided into G groups, with attention restricted within groups.Source: §3.4 · p. 5
  • β≈70% early-exit rates and ρ<3% sparsity confine heavy computation to ambiguous instances, while Equation 8 describes grouping, downscaling, and adaptive-efficiency gains.The structural cost reduction is represented by 1/(Gk^4), and the adaptive term by (1−β)ρ^2.Source: §3.4 · p. 5

4 Experiments

Experiments compare VideoMM with encoder-driven and MLLM-driven methods across long-video benchmarks, then examine adaptive computation and input-resolution choices. VideoMM combines accuracy and throughput gains, while its adaptive routing and downscaling settings expose practical accuracy–efficiency trade-offs.

4.1 Settings

  • Evaluations use three open-source MLLMs, a unified 8,192-token budget, and LongVideoBench, LVBench, and VideoMME, sampling up to 512 frames at 1 FPS.Source: §4.1 · pp. 5–6

4.2 Main Results

  • VideoMM leads in both accuracy and throughput, while encoder-driven methods favor throughput and MLLM-driven methods favor accuracy.The comparison spans multiple models and benchmarks.Source: §4.2 · p. 6, Table 1 · p. 6
  • On Qwen3-VL-8B and Qwen2.5-VL-7B, VideoMM scores 64.52 and 61.35 across three benchmarks, versus VidCom’s 55.93 and 52.32.Source: §4.2 · p. 7

4.3 Analysis Experiments

  • At 512 frames, VideoMM prioritizes accuracy (67.54) with 3.69 throughput, while VideoMM+ reaches 4.90 throughput with 66.87 accuracy; adaptive variants outperform non-adaptive VideoMM− on the trade-off.Source: §4.3 · p. 7, Figure 5 · p. 7
  • At 512 frames, VideoMM reaches 67.54% accuracy, with speedups of 4.3× over vanilla and 2.8× over FlexSelect; its throughput matches FlexSelect at 256 frames.Source: Figure 4 · p. 6, §4.3 · p. 7
  • In the downscale-factor comparison, k=3 yields accuracy 65.89 versus 67.54 at k=2, while throughput rises from 3.69 to 6.74.Source: §4.3 · p. 8
  • Early exits occur at 76%–83% for localized perception tasks, versus approximately 60% for relation-centric tasks requiring global temporal logic.The framework routes the latter to micro-level refinement when macro-level consensus diverges.Source: §4.3 · p. 8, Table 5 · p. 8

5 Conclusion

VideoMM addresses the accuracy–efficiency dilemma in long-video token reduction by using a Macro Proxy for semantic filtering and activating high-fidelity tokens only when needed. Evaluations report faster inference and higher accuracy than full-context baselines, with a superior Pareto frontier across variants.

5 Conclusion

  • VideoMM achieves a 6.13× speedup and improves accuracy by over 7% compared to full-context baselines.Evaluations span LongVideoBench and other benchmarks.Source: §5 · p. 8
  • VideoMM decouples semantic filtering from dense processing through a cost-effective Macro Proxy and recruits high-fidelity tokens only when necessary.Adaptive Macro-Micro Inference dynamically allocates those tokens.Source: §5 · p. 8
  • The VideoMM family establishes a superior Pareto frontier compared to prior arts across variants under the Macro-Micro design.The authors present this as empirical confirmation of the hierarchical perceptual granularity paradigm.Source: §5 · p. 8

A Additional Related Works.

The related work describes dynamic-resolution MLLMs as a basis for VideoMM’s coarse-to-fine approach, alongside several complementary strategies for long-context inference. These include cache compression, sparse attention, speculative decoding, and agent-based video understanding.

A Additional Related Works.

  • Dynamic-resolution MLLMs support arbitrary native-resolution patch-wise tokenization, which VideoMM uses to coordinate coarse-to-fine video inference.Earlier fixed-resolution approaches hindered comprehension of content with diverse semantic details.Source: A Additional Related Works · p. 10
  • KV-cache eviction compresses retained caches after token processing, while sparse attention selects a subset of tokens for computation; both are orthogonal to token reduction.Cache methods assess importance or allocate budgets [9] [11] [17] [43] [4] [10] [13], and SparseMM applies inter-head allocation to multimodal cache management [28]; sparse-attention methods retain the full cache [15] [18] [23] [35].Source: A Additional Related Works · p. 10
  • Speculative decoding uses a smaller model to draft outputs for verification by a larger model, and visual token reduction is described as orthogonal to that paradigm.SpecVLM [14] combines visual-token compression with smaller-model drafting and larger-model verification.Source: A Additional Related Works · p. 10
  • Agent-based approaches address long-video redundancy through semantic exploration, including multi-agent collaboration, shot-based reasoning, and hypothesis verification.The cited examples include LVAgent [5], VideoChat-A1 [31], VideoHV-Agent [32], and Symphony [37]; the passage suggests exploring synergy with token reduction.Source: A Additional Related Works · pp. 10–11

B Benchmark Details.

The evaluation details cover three long-video benchmarks and three dynamic-resolution MLLMs. The benchmarks span fine-grained retrieval, hour-long analysis, and multi-scale temporal understanding, while the model descriptions note capabilities relevant to video inputs.

B Benchmark Details.

  • LongVideoBench comprises 3,763 web-collected videos up to one hour and 6,678 human-annotated multiple-choice questions across 17 fine-grained categories.It includes a referring-reasoning task that requires retrieving and reasoning over relevant multimodal information from specified video contexts.Source: B Benchmark Details · p. 12
  • VideoMME contains 900 videos spanning 254 hours and 2,700 question-answer pairs across six visual domains, with videos from 11 seconds to one hour and multimodal inputs.Inputs include frames, subtitles, and audio.Source: B Benchmark Details · p. 12
  • LVBench targets real-world long-video understanding over videos spanning several hours, with tasks focused on comprehension and information extraction.The benchmark was created to address a gap in evaluating MLLMs on such videos.Source: B Benchmark Details · p. 12

C Model Details

  • The three evaluated models—Qwen2.5-VL-7B-Instruct, GLM4.1V-9B-Thinking, and Qwen3-VL-8B-Instruct—all support dynamic resolution.The model overview identifies Qwen2.5-VL’s dynamic frame-rate sampling and temporal updates, GLM’s dynamic-resolution video inputs, and Qwen3-VL’s time-, width-, and height-aware positional embeddings and timestamp alignment.Source: C Model Details · p. 12

D Accelerating Token Compression Using Lightweight MLLMs

This section explores two extensions to VideoMM: using a smaller MLLM for token selection and adding an Ultra-Macro stage before the standard Macro Proxy. Both adjust the accuracy–throughput trade-off, with reported gains in throughput alongside accuracy costs.

D Accelerating Token Compression Using Lightweight MLLMs

E Three-Stage Micro-Micro Exploration.

D Accelerating Token Compression Using Lightweight MLLMs

  • VideoMM-Lite raises throughput from 3.69 to 4.19 samples/mins while accuracy dips from 67.54% to 66.34% compared to standard VideoMM.The variant uses Qwen3-VL-4B for selection and Qwen3-VL-8B for final inference, avoiding additional training overhead.Source: D Accelerating Token Compression Using… · pp. 12–13

E Three-Stage Micro-Micro Exploration.

  • VideoMM-multi uses an aggressively downsampled Ultra-Macro Proxy to filter obvious non-essential regions before passing remaining candidates to the Macro level.The evaluation uses k=2 with Qwen3-VL-8B on LongVideoBench.Source: E Three-Stage Micro-Micro Exploration · p. 13

Related papers

Loading 2609.16722v1…