PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

Yuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong, Chang Xu

arXiv:2604.06912v1cs.CVcs.AI

TL;DR

Global high-resolution scaling burdens MLLMs with redundant tokens because it ignores query intent and spatial sparsity. Q-Zoom uses query-aware gating and self-distilled RoI extraction with alignment and targeted fine-tuning, yielding a dominant accuracy–efficiency frontier across tasks and models.

  • Problem

    Global resolution scaling indiscriminately processes redundant visual tokens, ignoring query intent and spatial sparsity while bottlenecking inference.

  • Method

    Q-Zoom combines a Dynamic Gating Network, self-distilled RoI proposals, consistency-aware routing labels, and spatio-temporal alignment for coarse-to-fine perception.

  • Results

    Q-Zoom establishes a dominant Pareto frontier, delivering 2.52× and 4.39× accelerations on Document & OCR and High-Resolution tasks while matching the baseline’s peak accuracy.

  • Takeaways & Limitations

    The framework provides a plug-and-play adaptive perception module whose improvements transfer to Qwen3-VL, LLaVA, and RL-trained thinking-with-image models.

Abstract

from arXiv · show

MLLMs require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global resolution scaling paradigms indiscriminately flood the quadratic self-attention mechanism with visually redundant tokens, severely bottlenecking inference throughput while ignoring spatial sparsity and query intent. To overcome this, we propose Q-Zoom, a query-aware adaptive high-resolution perception framework that operates in an efficient coarse-to-fine manner. First, a lightweight Dynamic Gating Network safely bypasses high-resolution processing when coarse global features suffice. Second, for queries demanding fine-grained perception, a Self-Distilled Region Proposal Network (SD-RPN) precisely localizes the task-relevant Region-of-Interest (RoI) directly from intermediate feature spaces. To optimize these modules efficiently, the gating network uses a consistency-aware generation strategy to derive deterministic routing labels, while the SD-RPN employs a fully self-supervised distillation paradigm. A continuous spatio-temporal alignment scheme and targeted fine-tuning then seamlessly fuse the dense local RoI with the coarse global layout. Extensive experiments demonstrate that Q-Zoom establishes a dominant Pareto frontier. Using Qwen2.5-VL-7B as a primary testbed, Q-Zoom accelerates inference by 2.52 times on Document & OCR benchmarks and 4.39 times in High-Resolution scenarios while matching the baseline's peak accuracy. Furthermore, when configured for maximum perceptual fidelity, Q-Zoom surpasses the baseline's peak performance by 1.1% and 8.1% on these respective benchmarks. These robust improvements transfer seamlessly to Qwen3-VL, LLaVA, and emerging RL-based thinking-with-image models. Project page is available at https://yuhengsss.github.io/Q-Zoom/.

I. INTRODUCTION

Q-Zoom addresses the query-level and spatial redundancies of global high-resolution processing with a query-aware, coarse-to-fine framework. Its lightweight routing, region extraction, alignment, and training strategies target improved accuracy–efficiency trade-offs.

  • Motivation: Global resolution scaling wastes computation by processing high-resolution detail for simple queries and flooding self-attention with irrelevant background tokens.The problem reflects both query-level intent and spatial sparsity.
  • Q-Zoom framework: Q-Zoom uses a Dynamic Gating Network and SD-RPN in intermediate feature space to address query-level and spatial redundancy.The framework operates in a single prefilling pass.
  • Adaptive routing: The gate bypasses high-resolution processing for queries answerable from coarse features, while complex queries activate targeted high-resolution RoI extraction.This routing is performed dynamically during inference.
  • Results: 2.52× and 4.39× inference accelerations are reported on Document & OCR and High-Resolution tasks, respectively.The reported evaluation describes a new accuracy–efficiency Pareto frontier.
  • Optimization: Q-Zoom trains its gate with consistency-aware samples and its SD-RPN with self-supervised tri-state distillation, avoiding human annotations and expensive RL pipelines.These strategies provide data-efficient optimization for the two modules.
  • Alignment and fine-tuning: Continuous spatio-temporal positional encoding and targeted Post-SFT fuse dense local RoIs with the coarse global layout.The design is intended to preserve spatial reasoning while adding local detail.

A. General Perception in MLLMs

MLLM perception has progressed from fixed low-resolution inputs toward high-resolution and query-aware designs. These approaches improve fine-grained perception but face token, latency, and training-cost trade-offs.

  • General perception: Fixed-resolution MLLM architectures produce a static visual-token budget, while low-resolution vision-encoder pretraining limits fine-grained perception.Examples include Q-Former compression and vision encoders pretrained at 224 × 224 or 336 × 336.
  • High-resolution approaches: Auxiliary high-resolution encoders and spatially partitioned image patches are two major strategies for addressing the resolution gap.Localized patches are encoded independently and concatenated before entering the LLM.
  • Query-aware perception: Query-aware perception first identifies task-relevant RoIs from coarse input and then re-encodes only those regions at higher resolution.Existing methods are categorized as training-free, SFT, or RL approaches.
  • Existing methods: Training-free methods require multiple prefilling passes or heavy autoregressive decoding to derive attention signals for RoI extraction.SFT methods instead teach explicit RoI prediction or external-tool calls.
  • Existing methods: RL-based optimization incurs high GPU-memory costs and training instability, while inference can shift computation to lengthy language-model reasoning.The passage also notes dependence on massive proprietary teacher models for reward signals.
  • MLLM pipeline: In LLaVA-style architectures, the vision encoder, projector, and LLM process visual and textual embeddings during prefilling before autoregressive response decoding.Prefilling is substantially faster than autoregressive decoding for an equivalent token count.

B. Adaptive Dynamic Gating Mechanism

Q-Zoom formulates high-resolution perception as conditional routing: a lightweight gate predicts whether coarse visual context suffices, using consistency-filtered supervision. At inference, the gate bypasses or triggers RoI refinement according to a confidence threshold.

  • Consistency-aware Training Sample Generation: Consistency-aware sample generation evaluates responses across increasing resolutions and retains only cases that fail at low resolution but succeed at high resolution.Unstable cases with the opposite pattern are discarded.
  • Consistency-aware Training Sample Generation: Filtered samples assign Need-Refine when the response is incorrect and No-Refine when it is correct.These binary targets teach whether additional local visual detail changes answer quality.
  • Dynamic gating network: The gating network reuses pretrained backbone layers and processes intermediate multimodal representations to keep routing computationally lightweight.The gate uses the final user-query token to capture query semantics before predicting refinement probability.
  • Dynamic gating network: The gate is trained with binary cross-entropy between predicted refinement probability and deterministic routing labels.The objective is Lgate = BCE(Y pred, Y label).
  • Inference routing: At inference, predictions below τgate bypass the RoI branch, whereas predictions at or above τgate trigger specialized RoI extraction.The threshold controls the accuracy–efficiency trade-off.

C. Self-Distilled Region Proposal Network

SD-RPN reuses frozen-backbone intermediate features to localize task-relevant regions, then re-encodes only the selected high-resolution crop and efficiently fuses it with cached coarse context.

  • Lightweight RoI Prediction via Branched Feature Reuse:: SD-RPN operates on frozen-backbone intermediate features to predict a dense spatial heatmap for task-relevant RoI localization.The branch uses transformer blocks initialized from backbone layers and parallels the gating network.
  • Lightweight RoI Prediction via Branched Feature Reuse:: The localized high-resolution RoI is re-encoded to provide fine-grained context for final response generation.This conditional refinement avoids scaling the entire image for complex queries.
  • Lightweight RoI Prediction via Branched Feature Reuse:: The prediction head reuses the final RPN block’s native projection matrices to compare the query-token state with dense visual features in a shared latent space.The resulting inner products form the spatial heatmap without introducing randomly initialized projection parameters.
  • Lightweight RoI Prediction via Branched Feature Reuse:: The predicted heatmap is converted into a smoothed binary spatial mask, whose foreground bounding box determines the localized crop.Sigmoid activation, reshaping, Gaussian smoothing, and thresholding produce the foreground used for cropping.
  • Lightweight RoI Prediction via Branched Feature Reuse:: Prefix KV-cache reuse preserves the coarse prefix through layer B, forwarding only new RoI and shifted user tokens through early layers before recombination.This bypasses redundant coarse-context re-encoding and accelerates secondary prefilling.

2) Training SD-RPN via Self-Distillation:

SD-RPN is trained by distilling denoised internal cross-modal attention into selective tri-state RoI labels, using only confident foreground and background tokens while ignoring ambiguous regions.

  • Training SD-RPN via Self-Distillation:: Internal MLLM cross-attention provides the visual-grounding signal used to supervise SD-RPN without external localization data.The training paradigm refines attention-derived signals into pseudo-labels.
  • Training SD-RPN via Self-Distillation:: Raw attention maps are denoised by removing sink tokens, which can receive disproportionate attention despite lacking semantic relevance.Sink tokens are identified through anomalously large feature-representation L2 norms.
  • Training SD-RPN via Self-Distillation:: The pseudo-label pipeline assigns high-confidence tokens to foreground or background and ignores ambiguous intermediate regions.Foreground is defined by a relative-attention threshold, while background tokens must lie outside the foreground bounding box and have low attention.
  • Training SD-RPN via Self-Distillation:: For multi-turn dialogues, terminal user-query states are aggregated with dense visual states to compute a multi-turn RoI map.The RPN is optimized with selective BCE over valid tokens.

D. Spatio-Temporal Alignment and Targeted Fine-Tuning

Q-Zoom aligns dense local RoI tokens with the coarse global layout and uses targeted Post-SFT to fuse dual-stream visual inputs without losing spatial context.

  • Spatio-Temporal Alignment: RoI extraction can detach fine-grained details from their broader spatial context, causing spatial misalignment in MRoPE-based MLLMs.This particularly harms tasks requiring global spatial reasoning, such as relative object placement.
  • Spatio-Temporal Alignment: Temporal shifting assigns RoI tokens an auxiliary layer, distinguishing them from coarse tokens that share the same spatial footprint.The offset is defined as t_roi = t_src + δ, with δ = min(H, W).
  • Spatio-Temporal Alignment: Spatial interpolation derives dense RoI position IDs from normalized source-image bounding-box coordinates.This preserves semantic localization across the denser H′ × W′ RoI grid.
  • Targeted Post-SFT: Targeted Post-SFT corrects contextual imbalance when concentrated local features distract the pretrained backbone from global context.The method uses contrastive hard-sample mining instead of generic multimodal fine-tuning.

A. Experiment Settings

Experiments evaluate Q-Zoom across Document & OCR and High-Resolution benchmarks, comparing accuracy and throughput against multiple resolution, heuristic, routing, and prior RoI baselines.

  • Benchmarks: The evaluation covers five Document & OCR datasets and three High-Resolution or Vision-Centric datasets.General QA benchmarks are additionally used for core-component ablations.
  • Comparisons: Q-Zoom is compared with direct resolution scaling, training-free heuristics, RL-based routing, and the preliminary SD-RPN framework across LLaVA and Qwen architectures.The comparisons include S2, ViCrop, AdaptVision, and SD-RPN.
  • Main Results: 7.2% average gain and approximately 1.5× faster throughput are reported for Q-Zoom versus ViCrop on LLaVA-1.5-7B.On Qwen2.5-VL-7B, Q-Zoom achieves a greater-than-10× speedup over AdaptVision under the stated evaluation alignment.
  • Main Results: Over 30% efficiency improvement and absolute accuracy gains from +3.2% to +5.1% are reported over preliminary SD-RPN on Qwen baselines.The gains span Qwen3-VL-4B and Qwen2.5-VL-3B.
  • Qualitative Results: Q-Zoom resolves tiny or obscured evidence more effectively than the Qwen2.5-VL-7B baseline in TextVQA and V* Bench examples.The SD-RPN localizes fine-grained evidence through RoI heatmaps, while the baseline suffers from resolution compression.
  • Accuracy-Efficiency Trade-off: 2.52× acceleration and 53.0% token reduction are achieved on Document & OCR while surpassing the baseline’s 85.9% peak at 1,024 tokens.On High-Resolution benchmarks, Q-Zoom reaches 66.7% with 576 tokens, 4.39× acceleration, and 73.2% token reduction.

C. Ablation Study

Ablations attribute Q-Zoom’s accuracy-efficiency gains to its upgraded SD-RPN, alignment and targeted SFT, dynamic gating, and consistency-aware training procedures.

  • Core Components: Removing strict token constraints and adding 33K high-resolution DocVQA samples improve both perceptual accuracy and inference throughput.These changes are evaluated in the upgraded SD-RPN configuration.
  • Core Components: Targeted SFT restores global spatial reasoning when dense local RoI tokens distract the model from coarse global inputs.It does not degrade foundational intelligence on General QA benchmarks.
  • Core Components: Nearly 30% relative throughput improvement is obtained when dynamic gating bypasses the RoI branch for simpler Document/OCR and General QA queries.On detail-heavy High-Resolution benchmarks, the gate consistently triggers the RoI branch to maintain peak perceptual accuracy.
  • RoI Strategies: Training-free attention and GroundingDINO RoI alternatives suffer from noisy localization, costly decoding, or weak semantic reasoning compared with SD-RPN.The attention strategy requires an autoregressive decoding pass, while GroundingDINO provides only marginal Document & OCR gains.
  • Architecture: B = 18 and R = 3 are selected because accuracy peaks at backbone split depth 18 and improves through three RPN layers before regressing at four.The selected configuration is used to balance efficiency and precision.
  • Data Efficiency: 77.7% average performance is achieved with only 10K self-distilled pseudo-labels, rising to 78.9% with the full 185K dataset.A 50K-pseudo-label model reaches 78.1%, comparable to the 78.0% GT-supervised model.
  • Pseudo-Labels: Equal foreground and background thresholds force ambiguous middle-range tokens into binary labels, producing a noticeable performance downgrade.The proposed tri-state assignment avoids this naive labeling configuration.
  • Consistency-Aware Gating: At an 85.5% accuracy threshold, consistency-aware gating bypasses RoI extraction for an additional 16.5% of queries over naive labeling.Multi-resolution consistency also produces faster convergence to a lower training-loss bound and strict Pareto dominance.

V. CONCLUSION

Q-Zoom addresses query-level and spatial redundancy by dynamically routing high-resolution processing and localizing task-relevant evidence in intermediate feature space.

  • Conclusion: Q-Zoom uses a Dynamic Gating Mechanism to bypass high-resolution processing for simple queries and an SD-RPN to localize evidence for detail-demanding tasks.Both lightweight modules operate during the initial prefilling stage.
  • Conclusion: A consistency-aware sample-generation strategy trains the gate, while self-supervised tri-state distillation trains SD-RPN without human annotations or external detection experts.The framework is designed to decouple perceptual fidelity from computational cost.

APPENDIX A IMPLEMENTATION AND PROMPT DETAILS

The appendix specifies Q-Zoom’s training settings, data usage, backbone configurations, and prompt variants for its core components. It also provides the associated tables and distinguishes natural-image prompting for Qwen models.

  • Training Configurations: All three Q-Zoom components use AdamW, zero weight decay, β1=0.9, β2=0.98, cosine decay, 3% warmup, and one training epoch.These settings are summarized in Table IX.
  • Dataset Usage and Filtering: The training mixture covers visual question answering, document and chart understanding, and fine-grained spatial reasoning datasets.Table X provides the component-level dataset and sample-size breakdown.
  • Backbone and Branch Configurations: All evaluated models use branch depth R = 3 for both SD-RPN and dynamic gating.Table XI reports the backbone split depth B for the evaluated models.
  • Backbone and Branch Configurations: Zoom-within-Zoom variants inherit the structural settings of their corresponding base models.
  • SD-RPN Prompts: LLaVA-1.5 and Qwen-series textual mode use a short-answer prompt, while Qwen natural-image mode uses a distinct prompt.The prompt distinction addresses substantially different Qwen attention distributions across textual/document and natural images.
  • SD-RPN Prompts: The natural-image prompt appends explicit grounding instructions to encourage spatially localized attention.

QWEN SD-RPN NATURAL MODE

The Qwen SD-RPN natural mode outputs a grounding box for the question’s RoI, while the Qwen Post-SFT judge compares base and RoI predictions using semantic correctness criteria.

  • QWEN SD-RPN NATURAL MODE: The natural-image SD-RPN prompt requests one raw-text bounding box per line for the question’s Region of Interest.The required format is x_min y_min x_max y_max followed by a detail label.
  • QWEN POST-SFT JUDGE PROMPT: The Post-SFT judge identifies cases where the base model succeeds but the RoI model fails by comparing both predictions with ground truth.This procedure is exclusive to the Qwen-series Post-SFT stage.
  • QWEN POST-SFT JUDGE PROMPT: The judge prioritizes semantic correctness over strict string matching in visual question answering.Its criteria accept contained values, standard synonyms, and formatting differences while rejecting factual errors or missing key information.
  • QWEN POST-SFT JUDGE PROMPT: The evaluation presents the question, ground truth, and predictions from the base and RoI models as the judge’s input data.
  • QWEN POST-SFT JUDGE PROMPT: The judge returns separate Yes/No correctness decisions for Model A and Model B in a fixed format.The required response is A: [Yes/No] B: [Yes/No].
Loading 2604.06912v1…