PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

Yuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong, Chang Xu

arXiv:2604.06912v1cs.CVcs.AI

TL;DR

Global high-resolution scaling burdens MLLMs with redundant tokens because it ignores query intent and spatial sparsity. Q-Zoom uses query-aware gating and self-distilled RoI extraction with alignment and targeted fine-tuning, yielding a dominant accuracy–efficiency frontier across tasks and models.

  • Problem

    Global resolution scaling indiscriminately processes redundant visual tokens, ignoring query intent and spatial sparsity while bottlenecking inference.

    Source: §I · p. 1

  • Method

    Q-Zoom combines a Dynamic Gating Network, self-distilled RoI proposals, consistency-aware routing labels, and spatio-temporal alignment for coarse-to-fine perception.

    Source: §I · pp. 1–2, D. Spatio-Temporal Alignment and… · p. 7

  • Results

    Q-Zoom establishes a dominant Pareto frontier, delivering 2.52× and 4.39× accelerations on Document & OCR and High-Resolution tasks while matching the baseline’s peak accuracy.

    Source: §I · p. 2, B. Main Results · p. 9

  • Takeaways & Limitations

    The framework provides a plug-and-play adaptive perception module whose improvements transfer to Qwen3-VL, LLaVA, and RL-trained thinking-with-image models.

    Source: §I · p. 2, B. Main Results · p. 9

Abstract

from arXiv · show

MLLMs require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global resolution scaling paradigms indiscriminately flood the quadratic self-attention mechanism with visually redundant tokens, severely bottlenecking inference throughput while ignoring spatial sparsity and query intent. To overcome this, we propose Q-Zoom, a query-aware adaptive high-resolution perception framework that operates in an efficient coarse-to-fine manner. First, a lightweight Dynamic Gating Network safely bypasses high-resolution processing when coarse global features suffice. Second, for queries demanding fine-grained perception, a Self-Distilled Region Proposal Network (SD-RPN) precisely localizes the task-relevant Region-of-Interest (RoI) directly from intermediate feature spaces. To optimize these modules efficiently, the gating network uses a consistency-aware generation strategy to derive deterministic routing labels, while the SD-RPN employs a fully self-supervised distillation paradigm. A continuous spatio-temporal alignment scheme and targeted fine-tuning then seamlessly fuse the dense local RoI with the coarse global layout. Extensive experiments demonstrate that Q-Zoom establishes a dominant Pareto frontier. Using Qwen2.5-VL-7B as a primary testbed, Q-Zoom accelerates inference by 2.52 times on Document & OCR benchmarks and 4.39 times in High-Resolution scenarios while matching the baseline's peak accuracy. Furthermore, when configured for maximum perceptual fidelity, Q-Zoom surpasses the baseline's peak performance by 1.1% and 8.1% on these respective benchmarks. These robust improvements transfer seamlessly to Qwen3-VL, LLaVA, and emerging RL-based thinking-with-image models. Project page is available at https://yuhengsss.github.io/Q-Zoom/.

I. INTRODUCTION

Q-Zoom addresses the query-level and spatial redundancies of global high-resolution processing with a query-aware, coarse-to-fine framework. Its lightweight routing, region extraction, alignment, and training strategies target improved accuracy–efficiency trade-offs.

Motivation

  • Global resolution scaling wastes computation by processing high-resolution detail for simple queries and flooding self-attention with irrelevant background tokens.The problem reflects both query-level intent and spatial sparsity.Source: §I · p. 1

Q-Zoom framework

  • Q-Zoom uses a Dynamic Gating Network and SD-RPN in intermediate feature space to address query-level and spatial redundancy.The framework operates in a single prefilling pass.Source: §I · pp. 1–2

Adaptive routing

  • The gate bypasses high-resolution processing for queries answerable from coarse features, while complex queries activate targeted high-resolution RoI extraction.This routing is performed dynamically during inference.Source: §I · p. 2

Results

  • 2.52× and 4.39× inference accelerations are reported on Document & OCR and High-Resolution tasks, respectively.The reported evaluation describes a new accuracy–efficiency Pareto frontier.Source: §I · p. 2

Optimization

  • Q-Zoom trains its gate with consistency-aware samples and its SD-RPN with self-supervised tri-state distillation, avoiding human annotations and expensive RL pipelines.These strategies provide data-efficient optimization for the two modules.Source: §I · p. 2

Alignment and fine-tuning

  • Continuous spatio-temporal positional encoding and targeted Post-SFT fuse dense local RoIs with the coarse global layout.The design is intended to preserve spatial reasoning while adding local detail.Source: §I · p. 2

A. General Perception in MLLMs

MLLM perception has progressed from fixed low-resolution inputs toward high-resolution and query-aware designs. These approaches improve fine-grained perception but face token, latency, and training-cost trade-offs.

General perception

  • Fixed-resolution MLLM architectures produce a static visual-token budget, while low-resolution vision-encoder pretraining limits fine-grained perception.Examples include Q-Former compression and vision encoders pretrained at 224 × 224 or 336 × 336.Source: A. General Perception in MLLMs · pp. 2–3

High-resolution approaches

  • Auxiliary high-resolution encoders and spatially partitioned image patches are two major strategies for addressing the resolution gap.Localized patches are encoded independently and concatenated before entering the LLM.Source: A. General Perception in MLLMs · p. 3

Query-aware perception

  • Query-aware perception first identifies task-relevant RoIs from coarse input and then re-encodes only those regions at higher resolution.Existing methods are categorized as training-free, SFT, or RL approaches.Source: B. Query-aware Perception in MLLMs · p. 3

Existing methods

  • Training-free methods require multiple prefilling passes or heavy autoregressive decoding to derive attention signals for RoI extraction.SFT methods instead teach explicit RoI prediction or external-tool calls.Source: B. Query-aware Perception in MLLMs · p. 3
  • RL-based optimization incurs high GPU-memory costs and training instability, while inference can shift computation to lengthy language-model reasoning.The passage also notes dependence on massive proprietary teacher models for reward signals.Source: B. Query-aware Perception in MLLMs · p. 3

MLLM pipeline

  • In LLaVA-style architectures, the vision encoder, projector, and LLM process visual and textual embeddings during prefilling before autoregressive response decoding.Prefilling is substantially faster than autoregressive decoding for an equivalent token count.Source: A. Preliminaries · p. 3

B. Adaptive Dynamic Gating Mechanism

Q-Zoom formulates high-resolution perception as conditional routing: a lightweight gate predicts whether coarse visual context suffices, using consistency-filtered supervision. At inference, the gate bypasses or triggers RoI refinement according to a confidence threshold.

Consistency-aware Training Sample Generation

  • Consistency-aware sample generation evaluates responses across increasing resolutions and retains only cases that fail at low resolution but succeed at high resolution.Unstable cases with the opposite pattern are discarded.Source: 1) Consistency-aware Training Sample… · p. 4
  • Filtered samples assign Need-Refine when the response is incorrect and No-Refine when it is correct.These binary targets teach whether additional local visual detail changes answer quality.Source: 1) Consistency-aware Training Sample… · p. 4

Dynamic gating network

  • The gating network reuses pretrained backbone layers and processes intermediate multimodal representations to keep routing computationally lightweight.The gate uses the final user-query token to capture query semantics before predicting refinement probability.Source: 2) Gating Network Architecture and… · p. 4
  • The gate is trained with binary cross-entropy between predicted refinement probability and deterministic routing labels.The objective is Lgate = BCE(Y pred, Y label).Source: 2) Gating Network Architecture and… · pp. 4–5

Inference routing

  • At inference, predictions below τgate bypass the RoI branch, whereas predictions at or above τgate trigger specialized RoI extraction.The threshold controls the accuracy–efficiency trade-off.Source: 2) Gating Network Architecture and… · p. 5

C. Self-Distilled Region Proposal Network

SD-RPN reuses frozen-backbone intermediate features to localize task-relevant regions, then re-encodes only the selected high-resolution crop and efficiently fuses it with cached coarse context.

Lightweight RoI Prediction via Branched Feature Reuse:

2) Training SD-RPN via Self-Distillation:

SD-RPN is trained by distilling denoised internal cross-modal attention into selective tri-state RoI labels, using only confident foreground and background tokens while ignoring ambiguous regions.

Training SD-RPN via Self-Distillation:

  • Internal MLLM cross-attention provides the visual-grounding signal used to supervise SD-RPN without external localization data.The training paradigm refines attention-derived signals into pseudo-labels.Source: 2) Training SD-RPN via Self-Distillation · p. 5
  • Raw attention maps are denoised by removing sink tokens, which can receive disproportionate attention despite lacking semantic relevance.Sink tokens are identified through anomalously large feature-representation L2 norms.Source: 2) Training SD-RPN via Self-Distillation · p. 6
  • The pseudo-label pipeline assigns high-confidence tokens to foreground or background and ignores ambiguous intermediate regions.Foreground is defined by a relative-attention threshold, while background tokens must lie outside the foreground bounding box and have low attention.Source: 2) Training SD-RPN via Self-Distillation · p. 6, Figure 4 · p. 6
  • For multi-turn dialogues, terminal user-query states are aggregated with dense visual states to compute a multi-turn RoI map.The RPN is optimized with selective BCE over valid tokens.Source: 2) Training SD-RPN via Self-Distillation · p. 7

D. Spatio-Temporal Alignment and Targeted Fine-Tuning

Q-Zoom aligns dense local RoI tokens with the coarse global layout and uses targeted Post-SFT to fuse dual-stream visual inputs without losing spatial context.

Spatio-Temporal Alignment

  • RoI extraction can detach fine-grained details from their broader spatial context, causing spatial misalignment in MRoPE-based MLLMs.This particularly harms tasks requiring global spatial reasoning, such as relative object placement.Source: D. Spatio-Temporal Alignment and… · p. 7
  • Temporal shifting assigns RoI tokens an auxiliary layer, distinguishing them from coarse tokens that share the same spatial footprint.The offset is defined as t_roi = t_src + δ, with δ = min(H, W).Source: D. Spatio-Temporal Alignment and… · p. 7
  • Spatial interpolation derives dense RoI position IDs from normalized source-image bounding-box coordinates.This preserves semantic localization across the denser H′ × W′ RoI grid.Source: D. Spatio-Temporal Alignment and… · p. 7

Targeted Post-SFT

  • Targeted Post-SFT corrects contextual imbalance when concentrated local features distract the pretrained backbone from global context.The method uses contrastive hard-sample mining instead of generic multimodal fine-tuning.Source: D. Spatio-Temporal Alignment and… · p. 7

A. Experiment Settings

Experiments evaluate Q-Zoom across Document & OCR and High-Resolution benchmarks, comparing accuracy and throughput against multiple resolution, heuristic, routing, and prior RoI baselines.

Benchmarks

  • The evaluation covers five Document & OCR datasets and three High-Resolution or Vision-Centric datasets.General QA benchmarks are additionally used for core-component ablations.Source: A. Experiment Settings · p. 7

Comparisons

  • Q-Zoom is compared with direct resolution scaling, training-free heuristics, RL-based routing, and the preliminary SD-RPN framework across LLaVA and Qwen architectures.The comparisons include S2, ViCrop, AdaptVision, and SD-RPN.Source: B. Main Results · p. 8

Main Results

  • 7.2% average gain and approximately 1.5× faster throughput are reported for Q-Zoom versus ViCrop on LLaVA-1.5-7B.On Qwen2.5-VL-7B, Q-Zoom achieves a greater-than-10× speedup over AdaptVision under the stated evaluation alignment.Source: B. Main Results · p. 8
  • Over 30% efficiency improvement and absolute accuracy gains from +3.2% to +5.1% are reported over preliminary SD-RPN on Qwen baselines.The gains span Qwen3-VL-4B and Qwen2.5-VL-3B.Source: B. Main Results · p. 8

Qualitative Results

  • Q-Zoom resolves tiny or obscured evidence more effectively than the Qwen2.5-VL-7B baseline in TextVQA and V* Bench examples.The SD-RPN localizes fine-grained evidence through RoI heatmaps, while the baseline suffers from resolution compression.Source: B. Main Results · p. 9

Accuracy-Efficiency Trade-off

  • 2.52× acceleration and 53.0% token reduction are achieved on Document & OCR while surpassing the baseline’s 85.9% peak at 1,024 tokens.On High-Resolution benchmarks, Q-Zoom reaches 66.7% with 576 tokens, 4.39× acceleration, and 73.2% token reduction.Source: B. Main Results · p. 9

C. Ablation Study

Ablations attribute Q-Zoom’s accuracy-efficiency gains to its upgraded SD-RPN, alignment and targeted SFT, dynamic gating, and consistency-aware training procedures.

Core Components

  • Removing strict token constraints and adding 33K high-resolution DocVQA samples improve both perceptual accuracy and inference throughput.These changes are evaluated in the upgraded SD-RPN configuration.Source: C. Ablation Study · p. 10
  • Targeted SFT restores global spatial reasoning when dense local RoI tokens distract the model from coarse global inputs.It does not degrade foundational intelligence on General QA benchmarks.Source: C. Ablation Study · p. 10
  • Nearly 30% relative throughput improvement is obtained when dynamic gating bypasses the RoI branch for simpler Document/OCR and General QA queries.On detail-heavy High-Resolution benchmarks, the gate consistently triggers the RoI branch to maintain peak perceptual accuracy.Source: C. Ablation Study · p. 10

RoI Strategies

  • Training-free attention and GroundingDINO RoI alternatives suffer from noisy localization, costly decoding, or weak semantic reasoning compared with SD-RPN.The attention strategy requires an autoregressive decoding pass, while GroundingDINO provides only marginal Document & OCR gains.Source: C. Ablation Study · p. 10

Architecture

  • B = 18 and R = 3 are selected because accuracy peaks at backbone split depth 18 and improves through three RPN layers before regressing at four.The selected configuration is used to balance efficiency and precision.Source: C. Ablation Study · pp. 10–11, Table VI · p. 11

Data Efficiency

  • 77.7% average performance is achieved with only 10K self-distilled pseudo-labels, rising to 78.9% with the full 185K dataset.A 50K-pseudo-label model reaches 78.1%, comparable to the 78.0% GT-supervised model.Source: C. Ablation Study · p. 11, Table VII · p. 11

Pseudo-Labels

  • Equal foreground and background thresholds force ambiguous middle-range tokens into binary labels, producing a noticeable performance downgrade.The proposed tri-state assignment avoids this naive labeling configuration.Source: C. Ablation Study · pp. 11–12, Table VIII · p. 12

Consistency-Aware Gating

  • At an 85.5% accuracy threshold, consistency-aware gating bypasses RoI extraction for an additional 16.5% of queries over naive labeling.Multi-resolution consistency also produces faster convergence to a lower training-loss bound and strict Pareto dominance.Source: C. Ablation Study · p. 12, Figure 9 · p. 12

V. CONCLUSION

Q-Zoom addresses query-level and spatial redundancy by dynamically routing high-resolution processing and localizing task-relevant evidence in intermediate feature space.

Conclusion

  • Q-Zoom uses a Dynamic Gating Mechanism to bypass high-resolution processing for simple queries and an SD-RPN to localize evidence for detail-demanding tasks.Both lightweight modules operate during the initial prefilling stage.Source: §V · p. 12
  • A consistency-aware sample-generation strategy trains the gate, while self-supervised tri-state distillation trains SD-RPN without human annotations or external detection experts.The framework is designed to decouple perceptual fidelity from computational cost.Source: §V · p. 12

APPENDIX A IMPLEMENTATION AND PROMPT DETAILS

The appendix specifies Q-Zoom’s training settings, data usage, backbone configurations, and prompt variants for its core components. It also provides the associated tables and distinguishes natural-image prompting for Qwen models.

Training Configurations

Dataset Usage and Filtering

Backbone and Branch Configurations

SD-RPN Prompts

  • LLaVA-1.5 and Qwen-series textual mode use a short-answer prompt, while Qwen natural-image mode uses a distinct prompt.The prompt distinction addresses substantially different Qwen attention distributions across textual/document and natural images.Source: B. Prompt Usage · p. 15, DEFAULT PROMPT · p. 15
  • The natural-image prompt appends explicit grounding instructions to encourage spatially localized attention.Source: B. Prompt Usage · p. 15

QWEN SD-RPN NATURAL MODE

The Qwen SD-RPN natural mode outputs a grounding box for the question’s RoI, while the Qwen Post-SFT judge compares base and RoI predictions using semantic correctness criteria.

QWEN SD-RPN NATURAL MODE

  • The natural-image SD-RPN prompt requests one raw-text bounding box per line for the question’s Region of Interest.The required format is x_min y_min x_max y_max followed by a detail label.Source: QWEN SD-RPN NATURAL MODE · p. 15

QWEN POST-SFT JUDGE PROMPT

  • The Post-SFT judge identifies cases where the base model succeeds but the RoI model fails by comparing both predictions with ground truth.This procedure is exclusive to the Qwen-series Post-SFT stage.Source: QWEN SD-RPN NATURAL MODE · pp. 15–16
  • The judge prioritizes semantic correctness over strict string matching in visual question answering.Its criteria accept contained values, standard synonyms, and formatting differences while rejecting factual errors or missing key information.Source: QWEN POST-SFT JUDGE PROMPT · p. 16
  • The evaluation presents the question, ground truth, and predictions from the base and RoI models as the judge’s input data.Source: QWEN POST-SFT JUDGE PROMPT · p. 16
  • The judge returns separate Yes/No correctness decisions for Model A and Model B in a fixed format.The required response is A: [Yes/No] B: [Yes/No].Source: QWEN POST-SFT JUDGE PROMPT · p. 16

Related papers

Loading 2604.06912v1…