PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Search

Yunqi Zhou, Chengjie Jiang, Chun Yuan, Jing Li

arXiv:2511.20460v2cs.CV

TL;DR

Ultra-HR remote-sensing imagery poses a mismatch for foundation models: resizing can lose answer-critical details, while full-image encoding strains token and memory budgets. ZoomSearch performs training-free, coarse-to-fine region search and layout-aware patch reassembly before answer prediction. Across two datasets and nine task types, it reports higher accuracy than competing methods and faster inference than existing methods.

  • Problem

    Remote-sensing foundation models struggle with Ultra-HR inputs because resizing can discard answer-critical detail and full-image processing can exceed token and memory budgets.

    Source: §1 · p. 2

  • Method

    ZoomSearch is a training-free, plug-and-play pipeline that uses Adaptive Multi-Branch Zoom Search to find query-relevant regions and Layout-Aware Patch Reassembly to form a compact canvas.

    Source: §4.2 · p. 4, §6 · p. 8

  • Results

    ZoomSearch+LLaVA-ov surpasses the best competing method by 12.5% on LRS-VQA and 6.3% points on MME-RealWorld-RS in average accuracy, across two datasets and nine task types.

    Source: §5.3 · p. 8

  • Takeaways & Limitations

    The experiments report accuracy gains and faster inference while preserving spatial cues under a low visual-token budget.

    Source: §6 · p. 8

    Evaluation identifies backbone-limited answering and imperfect search, including missed relevant instances or retained distractors.

    Source: S10.2. Failure Cases and Analysis · p. 13

Abstract

from arXiv · show

With advances in satellite constellations, sensor technologies, and imaging pipelines, ultra-high-resolution (Ultra-HR) remote sensing imagery is becoming increasingly widespread. However, current remote sensing foundation models are ill-suited to such inputs: full-image encoding exhausts token and memory budgets, while resize-based preprocessing loses fine-grained and answer-critical details. In this context, guiding the model look where it matters before prediction becomes crucial. Therefore, we present ZoomSearch, a training-free, plug-and-play pipeline that decouples 'where to look' from 'how to answer' for Ultra-HR Remote Sensing Visual Question Answering (RS-VQA). ZoomSearch combines Adaptive Multi-Branch Zoom Search, which performs a hierarchical search over image patches to localize query-relevant regions, with Layout-Aware Patch Reassembly, which reorganizes the selected patches into a compact, layout-faithful canvas. We conduct comprehensive experiments on Ultra-HR RS-VQA benchmarks MME-RealWorld-RS and LRS-VQA, comparing against (i) strong general foundation models, (ii) remote sensing foundation models, (iii) Ultra-HR RS-VQA methods, and (iv) plug-and-play search-based VQA methods. When integrated with LLaVA-ov, ZoomSearch attains state-of-the-art accuracy across diverse tasks, improving the LLaVA-ov baseline by 26.3% on LRS-VQA and 114.8% on MME-RealWorld-RS. Meanwhile, it achieves much higher inference efficiency, outperforming prior search-based methods by 20%~44% in speed.

1. Introduction

Ultra-HR imagery challenges remote-sensing foundation models because resizing can discard answer-critical detail, while encoding whole images can exceed token and memory budgets. ZoomSearch separates region search from answer generation and combines hierarchical search with layout-aware patch reassembly.

1. Introduction

  • Resize-based preprocessing loses fine structures and small objects, while tile-and-concatenate can require ∼300K visual tokens for an 8K image at VHM scale.The latter can cause out-of-memory failures and prohibitive inference costs.Source: §1 · p. 2
  • Pilot studies found hierarchical search superior across search units and relative-and-global-layout preservation best among reassembly strategies.The stated rationale is that preserving spatial cues at both local and global levels supports reliable Ultra-HR RS-VQA.Source: §1 · p. 2
  • The pipeline uses score-guided coarse-to-fine zooming with adaptive top-k expansion, then reassembles selected regions into a compact canvas preserving local and global spatial cues.Patch–text relevance and foundation-model forward confidence guide candidate ranking; the final canvas is submitted for answer prediction without fine-tuning or architectural changes.Source: §1 · p. 2
  • 26.3% improvement on LRS-VQA and 114.8% on MME-RealWorld-RS over the LLaVA-ov baseline; inference cost falls 20∼44% versus existing search-based pipelines.The gains are reported when ZoomSearch is plugged into LLaVA-ov.Source: §1 · p. 2
  • ZoomSearch frames “where to look” as an explicit search problem before answer generation, decoupling region selection from answering in a training-free, plug-and-play pipeline.The paper positions this approach as adapting existing remote-sensing foundation models to 2∼20K imagery.Source: §1 · p. 2

2. Related Work

Related work addresses Ultra-HR remote-sensing VQA through token pruning and retrieval, and high-resolution VQA through search-based methods. ZoomSearch is presented as a training-free, plug-and-play framework for remote-sensing imagery’s resolution, clutter, and object-scale variation.

2.1. Ultra-HR Remote Sensing VQA

  • Token-pruning methods compress inputs but may lose detail through pooling or downsampling; retrieval methods depend on external database coverage and quality.Text-aware localization can still make low-resolution pruning decisions, while many pruning pipelines require trainable controllers or backbone modifications.Source: §2.1 · p. 3
  • RAG methods retrieve image–text pairs or knowledge from external databases, but database coverage and quality constrain practical robustness and scalability.Examples include ImageRAG and RS-RAG [32].Source: §2.1 · p. 3

2.2. Search-Based VQA Methods

  • Natural-image search methods focus computation on query-relevant regions, but remote-sensing imagery’s higher resolution, clutter, and object-scale variation call for a more flexible strategy.The paper positions ZoomSearch as a training-free, plug-and-play framework for this setting.Source: §2.2 · p. 3

3. Pilot Study

The pilot study investigates which search unit and patch-reassembly strategy suit Ultra-HR RS-VQA, using an LRS-VQA subset and a fixed foundation-model backbone. Its findings inform the final ZoomSearch design.

3. Pilot Study

  • The pilot study compares search units and reassembly strategies to inform the design of a search-based Ultra-HR RS-VQA framework.Its two questions are what constitutes an appropriate search unit and how identified regions should be organized before answer prediction.Source: §3 · p. 3

3.1. Preliminary

  • Search-based approaches identify query-relevant regions before predicting answers from selected visual cues; pilot studies sample 2,000 LRS-VQA question–image pairs with max(H, W) ≥ 2K using llava-onevision-qwen2-0.5b.Source: §3.1 · p. 3

3.2. Exploring Proper Search Units

  • The hierarchical policy achieved the highest average performance, with particularly large gains on shape and status categories that depend on fine object geometry.The study interprets progressive coarse-to-fine zooming as better suited to multi-scale targets in Ultra-HR scenes.Source: §3.2 · p. 3

3.3. Exploring Reassembly Strategies

  • Relative-and-global-layout preserving reassembly achieved the highest accuracy on almost all categories among the tested strategies.It preserves mutual patch relations and their coarse locations in the original image, retaining spatial cues after reassembly.Source: §3.3 · p. 4, Figure 2 · p. 4

4. Method

ZoomSearch separates region localization from answer generation, then supplies the foundation model with selected patches composed into a compact, spatially faithful canvas. Its search and scoring mechanisms guide where to inspect and when to stop, while reassembly preserves local and global layout.

4.1. Method Overview

  • ZoomSearch replaces the full-resolution image with selected patches composed as compact, layout-faithful visual context for answer prediction.Adaptive Multi-Branch Zoom Search localizes informative patches before Layout-Aware Patch Reassembly organizes them.Source: §4.1 · p. 4

4.2. Adaptive Multi-Branch Zoom Search

  • Region localization is a tree search rooted at the whole image, with each patch split into a 3×3 grid and candidate patches scored by query relevance and model evidence.The search outputs a selected-patch set covering query-relevant regions.Source: §4.2 · p. 4
  • Search stops at minimum patch size, maximum depth, or sufficient evidence; otherwise, it retains threshold-passing children up to six and pursues multiple paths for dispersed targets.If no child passes the threshold, the best one is retained; search ends when the frontier empties or the global step budget is met.Source: §4.2 · p. 4

4.3. Layout-Aware Patch Reassembly

  • Reassembly anchors patches to one of nine coarse regions, then removes only empty rows and columns so their within-region order and arrangement remain intact.Each patch is assigned to a coarse region using its grid indices.Source: §4.3 · p. 5
  • A unified cell size and fixed 3Hc × 3Wc canvas preserve each region’s global position while compacting selected content for the foundation model.Patch blocks are resized with aspect-ratio-preserving padding and placed in their corresponding slots.Source: §4.3 · p. 6

4.4. Scoring Mechanism

  • Patch–text relevance averages retained tile scores after the lowest-scoring half is masked, while a binary-prompt confidence estimates whether the patch contains sufficient answer evidence.The final score increases the model-evidence contribution with search depth because early sufficiency judgments can be unreliable.Source: §4.4 · p. 6

5. Experiments

Experiments on LRS-VQA and MME-RealWorld-RS compare ZoomSearch with foundation models and search-based methods. ZoomSearch+LLaVA-ov leads average accuracy on both benchmarks, with faster inference than prior search-based methods and ablations supporting its scoring and branching choices.

5.1. Experimental Setting

  • The evaluation uses two benchmarks with open-ended or multiple-choice questions and compares five groups of foundation-model, Ultra-HR, and search-based baselines.For fair comparison, ZoomEye, RAP, and ZoomSearch use the same LLaVA-onevision-qwen2-7b base model.Source: §5.1 · p. 6

5.3. Quantitative Results

  • ZoomSearch+LLaVA-ov surpasses the best competing method by 12.5% on LRS-VQA and 6.3 percentage points on MME-RealWorld-RS average accuracy, without retraining or architectural changes.On LRS-VQA, it also reaches state of the art on global Rural/Urban classification and fine-grained object Category tasks.Source: Table 1 · p. 7, §5.3 · pp. 7–8, Table 2 · p. 8

5.2. Qualitative Results

  • On color recognition and counting examples, ZoomSearch reaches the target region and answers correctly, while the search-based baselines fail or include distractors.ZoomEye selects the full image in the counting example; RAP finds the target but also retrieves distractors.Source: §5.2 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7

5.3. Quantitative Results

  • ZoomSearch improves average accuracy over ZoomEye by 26.2% and RAP by 9.1%, while reducing per-sample latency by 44.6% and 20.2%, respectively.The comparison averages results over both datasets; the authors attribute the gains to multi-scale search and adaptive top-k retention.Source: Table 3 · p. 8, §5.3 · p. 8

5.4. Ablation Study

  • Ablations find that pruning non-salient areas for both patch–text relevance and model evidence gives the best average score, 31.21, versus 27.21 without pruning.Adaptive top-k branching also yields the best average accuracy, 31.21 (+3.18), by retaining children above a fixed normalized-score threshold.Source: Table 4 · p. 8, §5.4 · p. 8, Table 5 · p. 8

6. Conclusion

ZoomSearch separates region search from answer generation through coarse-to-fine localization and layout-aware patch reassembly. Experiments report accuracy gains and faster inference than existing methods.

6. Conclusion

  • ZoomSearch combines coarse-to-fine Adaptive Multi-Branch Zoom Search with Layout-Aware Patch Reassembly to preserve inter-patch relations and global orientation in a compact canvas.The conclusion attributes the benefits to focusing computation on relevant regions while retaining spatial cues under a low visual-token budget.Source: §6 · p. 8

S7. Pilot Study: Extended Setup and Analyses

Pilot studies on a 2,000-sample LRS-VQA subset compare search units and patch reassembly strategies using LLaVA-ov. Hierarchical search and relative-plus-global layout preservation perform best overall, with a category-level exception for Color.

Setup

Exploring proper search units

  • Hierarchical search raises average accuracy 4.0% over fine-tile while reducing latency 62.4%, and improves accuracy 7.6% over coarse-patch at moderate time cost.All three policies outperform the backbone’s 24.59% average accuracy.Source: S7. Pilot Study: Extended Setup and… · p. 9
  • Hierarchical search gains on Status and Shape, while coarse-patch leads on Color at 49.52%.The passage suggests coarser scales capture chromatic cues better, whereas excessive zooming may emphasize local textures and add noise.Source: S7. Pilot Study: Extended Setup and… · p. 9

Exploring reassembly strategies

  • Relative-plus-global-layout reassembly achieves the highest average accuracy at 28.38%, with gains across Reasoning, Status, Shape, Color, and Background.It anchors coarse regions to preserve global orientation and compresses within regions while maintaining relative layout.Source: S7. Pilot Study: Extended Setup and… · p. 9

S8. Algorithmic Details of ZoomSearch

ZoomSearch takes an image and query, scores patches, selects query-relevant regions, and reassembles them into a compact canvas for a foundation model. Its algorithm combines bounded, adaptive search with layout-preserving reconstruction.

S8. Algorithmic Details of ZoomSearch

S8.2. Hyper-parameters

  • Table 6 summarizes ZoomSearch hyper-parameters and defaults, while Table 7 specifies how minimum search unit umin varies by backbone and image size.Source: S8.2. Hyper-parameters · p. 9

S8.3. Patch Scoring

  • Patch scoring combines patch–text relevance and model-evidence cues, increasing the evidence cue’s contribution with search depth before sigmoid normalization and ranking.The normalized score is used for ranking and fixed-threshold adaptive branching.Source: S8.3. Patch Scoring · p. 10

S8.4. Adaptive Multi-Branch Zoom Search

  • Search begins with a 3×3 grid, expands the highest-scoring frontier, and stops patches at depth, size, or evidence thresholds.Adaptive branching retains children above a score threshold, falls back to the top child if none qualify, and caps retained branches and total steps.Source: S8.4. Adaptive Multi-Branch Zoom Search · p. 10

S8.5. Layout-Aware Patch Reassembly

  • Reassembly anchors selected patches to coarse regions, removes empty rows and columns without reordering, then places padded blocks into a fixed-slot canvas.This preserves global location and within-region spatial arrangement.Source: S8.5. Layout-Aware Patch Reassembly · p. 10

S8.6. Complexity Analysis

  • For a 20K image with umin=224, the stated search depth is five; traversal is bounded by Smax nodes, while reassembly is linear in selected cells.Per-node cost is driven by the scoring model, with a low-overhead pruned-view query to the foundation model.Source: S8.6. Complexity Analysis · p. 11

S9. Extended Experiments

Extended experiments use a 2,000-sample LRS-VQA subset and test component ablations, backbone compatibility, and scoring-model robustness. The joint pipeline improves average accuracy across backbones and scoring-model variants.

S9.1. Ablations on the Two Core Components

  • 24.22% average accuracy is achieved with both core components active, compared with gains of 12.52% for search alone and 13.74% for layout-aware reassembly alone.The authors describe the components’ strengths as complementary.Source: S9.1. Ablations on the Two Core… · p. 12

S9.2. Plug-and-Play Across Different Base Models

  • +24.22% average accuracy is observed for LLaVA-ov-7b, alongside improvements for GeoChat, VHM, Qwen2.5-VL-7b, and LLaVA-v1.6-7b.Gains are particularly pronounced in fine-grained categories; remote-sensing backbones show a mild decline on Rural/urban, which the authors associate with a possible compact-canvas distribution shift.Source: S9.2. Plug-and-Play Across Different… · p. 12

S9.3. Ablation on Scoring Models

  • All four scoring-model variants improve over the backbone by approximately 22%, and VisRAG has the highest average accuracy with modest margins over alternatives.The search and reassembly pipeline remains fixed across these variants.Source: S9.3. Ablation on Scoring Models · p. 12

S10. Additional Results

Additional examples show ZoomSearch localizing fine targets while preserving spatial layout; failure analysis identifies backbone-limited answering and incomplete search, motivating more robust coverage and verification.

S10.1. Additional Qualitative Examples

S10.2. Failure Cases and Analysis

  • Evaluation failures include backbone-limited answering and incomplete or imperfect search; original-quadrant anchors constrain counting errors but do not eliminate them.Source: S10.2. Failure Cases and Analysis · pp. 13–14
  • Future work targets better coverage–precision balance, patch-coverage self-verification, and scoring coupled to backbone representations to reduce imperfect-search error accumulation.Source: S10.2. Failure Cases and Analysis · p. 14

Related papers

Loading 2511.20460v2…