PaperlayerView exact revision on arXiv

Source-linked AI summary

The Earth in One Gaze: Training-Free Active Focus for UHR Remote Sensing Understanding

Yao Zhang, Pengyu Dai, Wei Guo, Jian Liang, Jian Song, Yafei Ou, Hongruixuan Chen, Naoto Yokoya

arXiv:2609.31747v1cs.CVeess.IV

TL;DR

UHR remote-sensing interpretation requires balancing local detail and scene context within limited MLLM visual-input budgets, while crop-based inspection can separate evidence from its surroundings. GazeEarth uses question-guided selection and a topology-preserving full-scene focus, improving benchmark-averaged accuracy over direct and overview answering across three benchmarks and four frozen backbones.

  • Problem

    UHR scenes exceed MLLMs’ visual-token limits, and the central question is whether the bottleneck is locating evidence or presenting it effectively in context.

    Source: §1 · p. 2

  • Method

    GazeEarth uses a frozen MLLM to select evidence from an indexed overview, then deterministically warps the original image to focus a fixed-size full-scene view for the same model to answer from.

    Source: §1 · pp. 2–3, §3.2 · p. 5, §3.3 · p. 5

  • Results

    Across three benchmarks and four frozen backbones, GazeEarth gains 4.6–9.4 percentage points over direct answering and 3.4–4.3 over overview answering.

    Source: §1 · pp. 2–3, §6 · p. 10

  • Takeaways & Limitations

    Frozen MLLMs can guide where to look, but answering depends on presenting selected evidence in context rather than as detached crops.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

Multimodal large language models (MLLMs) must balance local detail against scene context when interpreting ultra-high-resolution (UHR) remote sensing (RS) imagery within a limited visual-input budget. Existing selection-based methods either prune tokens and select patches through relevance scoring, or crop actively through repeated inspection. Neither strategy directly redistributes pixels within a continuous full-scene view: the first retains selected tokens or patches, and the second re-encodes a crop detached from its surroundings. Our pilot study finds that a frozen MLLM already produces useful question-guided spatial requests, yet crop-based inspection of the selected regions does not consistently improve its answers. We therefore formulate UHR understanding as a question of where to spend a fixed pixel budget. Based on this, we introduce GazeEarth, a simple-yet-effective training-free framework that couples question-guided region selection with full-scene foveated observation. The MLLM selects evidence cells from an indexed overview; a deterministic, topology-preserving warp resamples the original image onto a fixed-size canvas, enlarging their shared neighborhood while compressing the periphery; the same frozen model answers from this focused view, using at most two MLLM calls and no external selector or iterative search. Across three UHR remote sensing benchmarks and four frozen backbones, GazeEarth improves benchmark-averaged accuracy by 4.6 to 9.4 percentage points over direct answering and 3.4 to 4.3 over overview answering, outperforming task-trained methods. Our analyses show that existing MLLMs can guide where to look in UHR images on their own, and that what they can infer from the selected evidence depends on how that evidence is presented.

1 INTRODUCTION

UHR remote-sensing interpretation must balance fine detail with scene context under MLLMs’ visual-token limits, while crop-based approaches can detach evidence from its surroundings. GazeEarth instead couples question-guided region selection with a topology-preserving, full-scene focused view.

1 INTRODUCTION

  • Across three benchmarks and four frozen backbones, GazeEarth gains 4.6–9.4 percentage points over direct answering and 3.4–4.3 over overview answering.The framework allocates pixels to the selected neighborhood while retaining the full scene.Source: §1 · pp. 2–3
  • Frozen MLLMs provide useful spatial requests, but cropping selected regions does not consistently improve downstream answering.UHR scenes exceed typical MLLM visual-token limits, and crop-based views can detach local details from surrounding spatial context.Source: §1 · pp. 2–3
  • GazeEarth pairs a frozen MLLM’s explicit region selection with deterministic, topology-preserving focus on a fixed-size canvas, using at most two MLLM calls and no external selector or iterative search.The same model selects evidence from an indexed overview and answers from the resulting focused observation.Source: §1 · pp. 2–3

2 MOTIVATION: RETHINKING EVIDENCE PRESENTATION

The probes ask whether UHR reasoning is limited by evidence localization or by how selected evidence is presented. They find that frozen MLLMs can localize relevant regions, while isolated crops can hurt answering on contextually demanding tasks.

2 MOTIVATION: RETHINKING EVIDENCE PRESENTATION

  • The probes distinguish the ability to locate evidence from the way extracted evidence is presented for downstream reasoning.They examine both spatial requests and whether selected regions help when shown as crops.Source: §2 · p. 3

Finding 1: the answerer can already guide where to look

  • Image-conditioned MLLM requests achieve 77.74% benchmark-averaged alignment, exceeding Question-only MLLM at 42.82%, SigLIP2 at 43.09%, and RemoteCLIP at 37.20%.On 537 LRS-GRO questions without explicit location cues, center-hit rate is 64.99%, versus 52.33% for SigLIP2 and 37.99% for RemoteCLIP.Source: §2 · p. 3

Finding 2: showing the model the crop does not reliably help

  • On MME-RealWorld-RS, the selected isolated crop reduces accuracy by 9.2 percentage points versus Overview, from 49.01% to 39.81%.Providing a global thumbnail alongside the crop still leaves accuracy below Overview.Source: §2 · pp. 3–4

Implication

  • Localization identifies where to inspect, but cropping can remove surrounding landmarks needed for spatial reasoning.The proposed implication is to allocate more pixels to the target while preserving its continuous spatial connection to the periphery.Source: §2 · p. 4

3 GAZEEARTH

GazeEarth uses a frozen MLLM to select question-relevant evidence, reallocates a fixed canvas budget to that evidence while retaining scene topology, and answers from the resulting focused view.

3.1 THE PERCEPTION-ACTION LOOP

  • The perception-action loop uses the same frozen MLLM to request a region from an overview, foveate the original image within a fixed canvas budget, and answer from that view.The framework requires at most two MLLM calls per question.Source: §3.1 · p. 4, §3.4 · p. 6

3.2 QUESTION GUIDED GAZE

  • The MLLM selects at most two grid cells covering evidence for the query, and parsing pads selected cells with context and merges overlaps into a spatial request.An empty request retains the global view, while a successfully resolved box in the query bypasses model-guided selection.Source: §3.2 · p. 5

3.3 TOPOLOGY-PRESERVING FOVEATED MAPPING

  • The foveated mapping enlarges the shared focus neighborhood and compresses the periphery, while preserving coordinate order and keeping evidence connected to surrounding landmarks.The focus span is determined using an enlargement factor of α = 1.5 and a focus-fraction threshold of β = 0.55.Source: §3.3 · pp. 5–6

3.4 EVIDENCE-GROUNDED ANSWERING

  • The identical frozen MLLM receives the original query and focused canvas to produce its final, evidence-grounded response.The final stage completes the perception-action-perception sequence.Source: §3.4 · p. 6

4 EXPERIMENTS

GazeEarth improves UHR remote-sensing accuracy across benchmarks and frozen backbones, while analyses distinguish gains from full-scene focus, question-guided allocation, and pixel-budget normalization.

4.1 EXPERIMENTAL SETUP

  • Experiments use LRS-GRO, MME-RealWorld-RS, and XLRS-Bench, with Qwen3-VL-8B primary and cross-backbone evaluation on LLaVA-v1.6-7B and Intern-S1-mini.The default setup uses a 4×4 indexed overview, selects up to two regions, and caps answer-stage views at 2,048 pixels on the long side.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • GazeEarth reaches 61.00% on LRS-GRO, 52.43% on MME-RealWorld-RS, and 53.51% on XLRS-Bench, improving over direct Qwen3-VL-8B inference by 4.10, 10.32, and 10.59 points.The main results tables report LRS-GRO and MME-RealWorld-RS scores in Table 3 and XLRS-Bench category scores in Table 4.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 8
  • Benchmark-averaged gains over Direct are 8.34, 9.37, 4.60, and 5.45 points for Qwen3-VL-8B, LLaVA-v1.6-7B, Intern-S1-mini, and GPT-4o, respectively.Gains over Overview are 4.27, 3.41, 3.78, and 3.58 points, respectively, with parameters unchanged.Source: §4.2 · pp. 8–9, Table 5 · p. 8
  • With Qwen3-VL-8B answering, GazeEarth reaches a 55.65% benchmark average, compared with 51.04% for SigLIP2 and RemoteCLIP and 50.01% for Random Focus.Self-directed gaze has a mean advantage of 4.61 points over both external observers.Source: Table 6 · p. 8, §4.2 · p. 9
  • At the same canvas and pixel budget, GazeEarth scores 55.65% in 5.07 seconds, versus Overview’s 51.38% in 4.02 seconds and Direct’s 47.31% in 7.34 seconds.Of GazeEarth’s 8.34-point improvement over Direct, 4.07 points come from canvas normalization and 4.27 from directed focus.Source: §4.2 · p. 9

4.3 ABLATION STUDY

  • Across presentation variants, GazeEarth averages 55.65%, versus 51.38% for Overview, 49.76% for Gaze + Crop, and 51.40% for Think with Image.On MME-RealWorld-RS, continuous full-scene focus scores 52.43%, versus 39.81% and 42.24% for the two crop-based variants.Source: §4.3 · p. 9, Table 7 · p. 9

5 RELATED WORK

Prior UHR remote-sensing approaches include token or patch selection and active inspection, but these strategies face challenges in retaining spatial context. GazeEarth instead uses an explicit spatial request to guide deterministic warping of the original image into a full-scene canvas.

Remote Sensing Multimodal Large Language Models

  • Remote-sensing MLLMs use descriptive and geographic supervision, grounded dialogue, multisensor alignment, and geospatial reasoning, while UHR methods include token retention, patch search, and local-global integration.The cited approaches are described as relying heavily on patch selection and struggling to preserve spatial context.Source: §5 · p. 9

Active Vision in General Domains and Remote Sensing

  • Active-vision methods include language-guided search, cropping, learned inspection, spatially arranged crops, and hierarchical evidence; some remote-sensing methods require post-training or repeated crop-and-integration rounds.GazeEarth elicits an explicit spatial request through text generation to guide deterministic piecewise-linear mapping of the original UHR image into a fixed-size full-scene canvas.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents GazeEarth as a training-free, full-scene focused-observation framework and reports gains over direct and overview answering across benchmarks and frozen backbones. The supplementary implementation passages describe its prompts, image handling, and inference protocol.

6 CONCLUSION

  • Across three benchmarks and four frozen backbones, GazeEarth improves benchmark-averaged accuracy by 4.6–9.4 points over direct answering and 3.4–4.3 points over overview answering.The conclusion frames effective observation construction, rather than repeated visual search alone, as a promising direction for UHR MLLM reasoning.Source: §6 · p. 10

A EXPERIMENTAL AND IMPLEMENTATION DETAILS

  • The supplementary analyses separate whether spatial requests are useful, how to render them for answering, and the accuracy–cost tradeoffs of the resulting pipeline.Comparisons are described as controlled where possible, with end-to-end gains not attributed to one component without a corresponding control.Source: A EXPERIMENTAL AND IMPLEMENTATION… · p. 14

A.1 PROMPTS

  • The same model first selects regions from an indexed overview and then answers from a resulting full-scene observation in a separate interaction.The answering call receives the original question and rendered view as a fresh interaction.Source: §A.1 · p. 14

A.2 BACKBONE INTERFACES AND INFERENCE PROTOCOL

  • Inference keeps backbone parameters fixed; the selector’s request shapes the rendered view, while the answerer receives that view and question without selector history.The overview uses a 4 × 4 grid, selects up to two cells, and the answer-stage canvas has a 2,048-pixel long-side cap.Source: §A.2 · p. 15

A.3 BENCHMARKS AND EVALUATION PROTOCOLS

The appendix defines benchmark and timing protocols, then examines whether frozen-model spatial requests align with question-relevant regions and how alignment relates to answering.

A.3 BENCHMARKS AND EVALUATION PROTOCOLS

  • The three benchmarks cover open-ended and multiple-choice remote-sensing QA, with categories spanning global, region, object, color, counting, position, perception, and reasoning.LRS-GRO accuracy uses answer matching or a WordNet path-similarity threshold of 0.8; the other benchmarks use multiple-choice matching, and cross-benchmark means are unweighted.Source: §A.3 · p. 16

A.4 HARDWARE AND TIMING PROTOCOL

  • Timing is measured per question on one NVIDIA A800 after image loading, including view construction, preprocessing, and answer generation but excluding initialization and source-image I/O.No dedicated warm-up or repeated timing trials are used.Source: §A.4 · p. 17

B.2 PLACEMENT RULE FOR THE TARGET INTERVAL

  • The foveated mapping places the enlarged focus interval on the canvas while retaining peripheral segments, then bilinearly resamples the original image into the final view.The coordinate mapping is monotonic along each axis and has no folds, preserving connectivity and horizontal and vertical order.Source: B FOVEATED MAPPING DETAILS · p. 17, §B.2 · pp. 17–18, §B.3 · p. 18

B.1 PARSING AND COORDINATE RULES

  • The indexed grid maps selected cells to source-image rectangles; padding adds neighboring context, overlapping regions merge, and a supplied labeled box maps directly to source coordinates.A resolved box is expanded to a contextual square; otherwise, the indexed-overview selector supplies the request.Source: §B.1 · p. 17

C.1 SETUP

  • The localization probe compares image-conditioned requests with external observers and question-only MLLM requests against question-derived reference cells; the presentation probe compares overview, crop, and unlinked overview-plus-crop views.Reference-cell statistics include only questions with a matched reference and extractable selected cell; d is the minimum Manhattan distance between selected and reference cells.Source: §C.1 · p. 18

C.2 SPATIAL-REQUEST ALIGNMENT AND TARGET LOCALIZATION

  • Image-conditioned MLLM requests achieve 77.74% mean alignment, versus 43.09% for SigLIP2, 37.20% for RemoteCLIP, and 42.82% for question-only MLLM.At r = 1, hit rates reach 89.49% on XLRS-Bench, 93.65% on LRS-GRO, and 93.33% on MME-RealWorld-RS; Figure 6 plots alignment by allowed grid distance.Source: §C.2 · pp. 18–19, Table 10 · p. 19, Figure 6 · p. 19

C.3 LOCALIZATION AGAINST INDEPENDENT TARGET ANNOTATIONS.

  • On 537 LRS-GRO questions without explicit location cues, GazeEarth reaches 64.99% center-hit and 55.65% mean target-box coverage.Its center-hit rate exceeds SigLIP2 and RemoteCLIP by 12.66 and 27.00 percentage points; oracle-bbox Gaze achieves 70.20% answering accuracy.Source: §C.3 · pp. 19–20, Table 11 · p. 20

C.4 ALIGNED VERSUS OFF-TARGET REQUESTS

  • For the MLLM’s own requests, aligned examples have 50.49% mean answer accuracy, compared with 42.00% for off-target examples.Alignment does not ensure a correct answer; these conditional accuracies combine evidence perception and answer reasoning.Source: §C.4 · p. 20

C.5 EVIDENCE PRESENTATION

The evidence-presentation probe finds that the effect of question-selected crops varies by benchmark and does not improve the three-benchmark mean. Extended results also report category-wise gains over Direct.

C.5 EVIDENCE PRESENTATION

  • 50.55% versus 45.13% on XLRS-Bench: the selected crop raises accuracy, and adding an overview raises it further to 51.43%.These are the Overview, Gaze + Crop, and Unlinked Global + Crop scores, respectively.Source: §C.5 · p. 20
  • 39.81% versus 49.01% on MME-RealWorld-RS: isolated cropping reduces accuracy, while adding an overview partly recovers it to 42.24%.These are the Gaze + Crop, Overview, and Unlinked Global + Crop scores, respectively.Source: §C.5 · p. 20
  • Across benchmarks, Overview, Gaze + Crop, and Unlinked Global + Crop average 51.38%, 49.76%, and 51.40%, respectively; crop presentation does not improve the three-benchmark mean.The crop helps on XLRS-Bench but hurts on MME-RealWorld-RS.Source: §C.5 · pp. 20–21

D EXTENDED QUANTITATIVE RESULTS

D.1 CATEGORY-LEVEL BREAKDOWN

Category-level comparisons show gains over Direct alongside substantial variation by benchmark and question type; no single evidence presentation leads every category.

D.1 CATEGORY-LEVEL BREAKDOWN

  • GazeEarth improves all reported categories over Direct across the three benchmarks; on XLRS-Bench, its largest gains are 25.00 points for Scene Classification and 15.00 for Planning.Counting gains differ by benchmark: MME-RealWorld-RS rises from 21.35% to 21.78%, while XLRS-Bench rises from 32.50% to 44.38%.Source: §D.1 · p. 21, Table 15 · p. 22
  • GazeEarth leads overall on XLRS-Bench and leads six listed categories, while Linked Global + Crop leads Object Properties and Random Focus leads Anomaly Reasoning.On LRS-GRO, GazeEarth leads Global, Object, and overall scores, while Linked Global + Crop leads Region questions.Source: §D.1 · pp. 21–22, Table 15 · p. 22, Table 16 · p. 22
  • On MME-RealWorld-RS, Gaze + Crop raises Color accuracy from 55.54% to 62.47% but lowers Position accuracy from 72.39% to 34.45%.The effects of evidence presentation vary across question categories.Source: §D.1 · p. 22, Table 17 · p. 22

D.2 OBSERVER SUBSTITUTION PER BENCHMARK

Observer substitution favors self-directed gaze overall, while deeper recursive gaze reduces both accuracy and efficiency; random focus falls below the overview mean.

D.2 OBSERVER SUBSTITUTION PER BENCHMARK

  • Self-directed gaze reaches 52.43% on MME-RealWorld-RS, 61.00% on LRS-GRO, and 53.51% on XLRS-Bench, exceeding the strongest external observer in each reported overall comparison.The respective margins are 4.94 points over SigLIP2, 0.90 over RemoteCLIP, and 7.41 over SigLIP2.Source: §D.2 · pp. 22–23, Table 18 · p. 23, Table 19 · p. 23

D.3 EFFECT OF GAZE DEPTH

  • A single gaze achieves 55.65% benchmark-averaged accuracy, versus 52.91% for Two-Gaze and 52.72% for Three-Gaze, while mean runtime rises from 5.07 to 5.95 and 7.02 seconds.Single gaze also has the highest overall accuracy on each benchmark among the tested depth variants; repeated transformations compose on the current view rather than accumulating observations.Source: §D.3 · p. 23

D.4 RANDOM FOCUS VERSUS OVERVIEW

  • Self-directed gaze reaches 55.65% mean accuracy, 4.27 points above Overview’s 51.38%; SigLIP2 and RemoteCLIP each reach 51.04%, while random selection reaches 50.01%.Random Focus falls below Overview on LRS-GRO and MME-RealWorld-RS but is slightly higher on XLRS-Bench.Source: §D.4 · pp. 23–24

E STATISTICAL ANALYSIS

The section tests GazeEarth’s accuracy gains statistically and outlines its inference-runtime components. The added cost over Overview comes mainly from selection.

E STATISTICAL ANALYSIS

  • pHolm < 0.001 for GazeEarth versus Direct on all datasets; versus Overview, gains are significant on MME-RealWorld-RS and XLRS-Bench, but not LRS-GRO.The LRS-GRO gain of +1.01 points is comparatively marginal after adjustment (pHolm = 0.078).Source: Table 21 · p. 24, E STATISTICAL ANALYSIS · p. 24

F EFFICIENCY ANALYSIS; F.1 MODEL CALLS AND OBSERVATION CONSTRUCTION.

  • Runtime breakdown separates selection, visual construction, and answering; selection contributes most of the additional cost over Overview.GazeEarth uses one selection call and one answering call, with deterministic resampling between them.Source: F EFFICIENCY ANALYSIS · p. 24, §F.1 · p. 24

F.2 ANSWERING COST.

The section reports answering-stage runtime comparisons and sensitivity to selection-grid size, region budget, and focus scale. The reported default settings include a 4 × 4 grid, at most two cells, and α = 1.5.

F.2 ANSWERING COST.

  • Answer generation takes 3.907 seconds on MME-RealWorld-RS and 3.800 seconds on LRS-GRO, saving 3.656 and 2.617 seconds versus Direct, respectively.These savings exceed the combined selection and visual-construction costs of 1.366 and 1.335 seconds, respectively.Source: §F.2 · pp. 24, 26

G.1 SELECTION-GRID SIZE.

  • 55.65% benchmark-averaged accuracy for a 4 × 4 grid exceeds 53.87% for 3 × 3 and 53.97% for 5 × 5.The 4 × 4 setting also gives the highest overall score on each benchmark among the three grid settings.Source: §G.1 · p. 26

G.2 REGION BUDGET.

  • 54.41% mean accuracy with two selected cells exceeds 54.41% with one?Source: §G.2 · p. 26

G.3 FOCUS SCALE.

  • 55.65% mean accuracy at α = 1.5 exceeds 53.92% at 1.25 and 54.51% at 1.75; on MME-RealWorld-RS, α = 1.75 improves Color and Count but lowers Position from 71.12% to 68.10%.Source: §G.3 · p. 26

G.4 SELECTOR RESOLUTION.

The 1536-pixel overview has the highest mean accuracy across the three benchmarks. The qualitative examples trace how observers select regions and how the resulting views shape answering, while the broader-impact discussion identifies implications for observation design and oversight.

G.4 SELECTOR RESOLUTION.

  • 55.65% mean accuracy at 1536 pixels, versus 54.31% at 1024 pixels and 54.68% at 2048 pixels, is the highest across the three benchmarks.MME-RealWorld-RS rises from 52.43% to 52.84% at higher overview resolution, while LRS-GRO falls from 61.00% to 60.57%.Source: §G.4 · p. 27

H QUALITATIVE EXAMPLES

  • The examples span spatial responses, region selection, and resulting visual observations across methods.The figures connect where observers look with their answers for the same image–question pairs.Source: H QUALITATIVE EXAMPLES · p. 27, §H.1 · p. 27

H.1 SPATIAL RESPONSE VISUALIZATION

  • Figures 9–14 pair spatial-response maps and cell summaries with recorded selections and answers, distinguishing the selected region from the response distribution visualized over the scene.Random Focus selects a cell without a learned response.Source: §H.1 · p. 28, Figure 9 · p. 28, Figure 10 · p. 28

H.2 OBSERVATION AND ANSWERING EXAMPLES

  • Figures 15–20 compare methods on object attributes, counting, and spatial relations, showing selected neighbourhoods enlarged alongside surrounding landmarks in one full-scene view.Spatial preservation retains region ordering and connectivity, though pixel coordinates and local scales change.Source: §H.2 · p. 30, Figure 15 · p. 30

I BROADER IMPACT

  • GazeEarth uses frozen, publicly available MLLMs without training or fine-tuning, and its explicit cell list and rendered view can be inspected for oversight and error auditing.In the Qwen3-VL-8B setting, the bounded answer-stage canvas yields 1.45× faster inference than native-resolution Direct inference.Source: I BROADER IMPACT · p. 33
  • The broader-impact discussion presents observation construction as a complementary design axis: frozen MLLMs can express where they need detail, and presentation affects what they can answer.The paper frames this as a route alongside larger encoders, longer visual contexts, and additional training.Source: I BROADER IMPACT · p. 33

Related papers

Loading 2609.31747v1…