Source-linked AI summary
From Coordinates to Candidate Regions: Temporal Change Localization via Region Selection in Remote Sensing Multimodal LLMs
Juwan Chung, Sungjune Park, Yeongyun Kim, Yong Man Ro
TL;DR
RS-MLLMs struggle to localize small, dense, and multiple changed regions with coordinate-token generation. This paper instead proposes text-conditioned candidate regions represented by spatially and temporally enriched tokens, with the LLM selecting regions directly; it reports stronger temporal localization and grounding while retaining competitive understanding performance.
Problem
RS-MLLM coordinate generation is fragile for small, densely packed objects and multiple changed regions, motivating a different localization formulation.
Method
The method uses text-conditioned region proposals, per-frame candidate tokens with spatial and temporal cues, and LLM selection of region tokens.
Results
Region selection substantially outperforms coordinate-generation baselines on temporal change localization and improves visual grounding while maintaining competitive understanding performance.
Takeaways & Limitations
Oracle analysis separates proposer recall from selector performance, providing a diagnostic basis for identifying whether candidate coverage or selection is limiting results.
Takeaways & Limitations
The method is limited by its candidate-proposal ceiling and currently operates at the axis-aligned-box level with limited support for long-sequence temporal reasoning.
Abstract
from arXiv · showhide
Remote sensing multimodal large language models (RS-MLLMs) have advanced scene understanding and visual question answering over satellite imagery, yet localizing specific objects or changed regions remains challenging. Existing approaches rely on generating bounding box coordinates as token sequences, which is fragile for the small, densely packed objects common in remote sensing and increasingly error-prone when multiple targets must be localized simultaneously. In this work, we present an RS-specific formulation of the region selection paradigm, previously explored in natural-image MLLMs, and extend it to temporal change localization over multi-image sequences. Our framework employs a text-conditioned region proposal module, encodes each candidate as special tokens carrying per-frame visual features enriched with spatial and temporal cues, and lets the LLM localize targets by selecting region tokens in its response. We construct a multi-task training and evaluation suite spanning localization, referring expression, visual grounding, and understanding tasks across single-image and multi-temporal settings. Experiments show that our approach substantially outperforms coordinate-generation baselines on temporal change localization, while improving single-image visual grounding and maintaining competitive understanding performance. Oracle analysis decomposes the contributions of the region proposer and the LLM selector, providing diagnostic insight unique to this framework. Our code will be available at https://github.com/juwan-kr/RS-RegionSelect.
1 Introduction
The paper addresses fragile coordinate-based localization in remote sensing by extending region selection to dense and temporal localization, supported by a multi-task evaluation suite.
- Motivation: Coordinate generation is fragile for small, densely packed objects and increasingly error-prone when multiple changed regions must be localized.Small coordinate errors can substantially degrade localization, while long multi-region coordinate sequences risk omission, duplication, and repetition.
- Approach: The framework extends region selection from natural-image MLLMs to remote sensing and temporal image sequences.It represents candidates with per-frame visual features and lets the LLM select relevant region tokens instead of generating coordinates.
- Evaluation: The multi-task suite evaluates localization, referring expression, grounding, and understanding across single-image and multi-temporal settings.Controlled ablations and oracle analysis separately quantify proposer and selector contributions.
- Approach: Each candidate carries aligned per-frame visual features for region-level cross-frame comparison during temporal change localization.This formulation is designed for dense localization and multi-image change detection.
- Approach: The region proposal module unifies heterogeneous detection datasets and filters classes using the natural-language query.Prompt-aware class filtering bridges the LLM query and detector vocabulary.
2 Related Work
Prior remote sensing MLLMs support conversational, temporal, grounding, and scene-understanding tasks, while this work combines region-level candidate selection with an RS framework for localization.
- Remote sensing MLLMs: Remote sensing MLLMs have expanded from conversational interaction to temporal reasoning, change localization, and multi-image temporal question answering.Related systems also address multi-sensor inputs, pixel-level grounding, cross-sensor fusion, and scene understanding.
- Region-level grounding: Visual grounding in MLLMs commonly uses coordinate generation or segmentation masks, alongside approaches that inject region-level features into the LLM.Examples include ROIAlign spatial tokens, dedicated region proposers, and retrieval-based region perception.
- Positioning: The framework’s region tokens connect the region proposal process with LLM localization responses.The overview shows candidate generation, feature projection, and response-time selection of tokens such as <roi4> and <roi6>.
3 Method
The method generates text-conditioned candidate regions, enriches each candidate with spatial and temporal visual tokens, and trains the LLM to select regions in its response across multi-temporal and single-image tasks.
- Task formulation: Region selection replaces coordinate tuples with single candidate tokens, reducing multi-target output length while injecting region-level visual information into the LLM context.The proposal module first produces candidates, and the LLM generates their corresponding tokens within a natural-language response.
- Region proposal: Text-conditioned open-vocabulary detection bridges natural-language queries to heterogeneous remote-sensing detector vocabularies.Inference ranks class names by semantic similarity, then deduplicates, applies NMS, and caps the candidate set.
- Region token representation: Each candidate contains a proxy token and one visual feature token per input frame.For bi-temporal tasks, the two visual slots expose the same spatial region before and after an event.
- Region token representation: Spatial embeddings encode candidate geometry, while temporal embeddings identify the source frame of each visual feature.The geometry descriptor includes box coordinates, center, width, height, area, and aspect ratio; LayerNorm and learnable scalars enrich the embeddings.
- LLM localization: The LLM receives image patches, candidate token groups, and the text query, then maps generated region tokens back to candidate bounding boxes.Per-frame candidate features allow comparison of the same spatial region across time within the LLM context.
- Training: Training spans multi-temporal change localization, referring, QA, temporal reasoning, scene classification, and single-image tasks.The dataset design combines temporal and single-image sources to support localization and broader remote-sensing understanding.
- Training: Training uses staged domain adaptation, region-module alignment, and subsequent optimization across the introduced components.Stage 1 adapts the model to scene-level remote-sensing tasks, while Stage 2 introduces the proposal module and trains new region-token parameters with the LLM frozen.
4 Experiments
Across localization, grounding, understanding, ablation, and diagnostic analyses, region selection consistently improves dense and temporal RS localization while preserving competitive understanding performance.
- Localization Tasks: 69.4% F1 vs. 32.3% for Ovis2.5-FT on xBD building localization, a 37.1-point improvement under the same base model, data, and recipe.On S2Looking change detection localization, the corresponding scores are 50.5 vs. 36.4.
- Localization Tasks: 77.3% accuracy vs. 69.5% for Ovis2.5-FT on DIOR-RSVG shows that region-selection benefits extend to single-image visual grounding.SRE also improves on xBD and S2Looking, although absolute scores remain moderate.
- Localization Tasks: The model outperforms TEOChat on all localization benchmarks, while qualitative examples show more reliable region identification than coordinate-generation baselines.On LEVIR-MCI and zero-shot HRCUS-CD, competing models produce degenerate, fragmented, or mislocated outputs, whereas the proposed model selects relevant regions.
- Understanding Tasks: Scene-level understanding remains comparable to TEOChat, while QFabric temporal tasks reach the best results: 78.0% TRE and 78.6% RTQA.The model scores 85.9 vs. 89.9 on xBD QA and 74.0 vs. 73.4 on S2Looking QA.
- Ablation Study: Removing the region module reduces S2Looking CDL from 50.5 to 36.4 and DIOR-RSVG from 77.3 to 69.5, isolating the localization formulation’s contribution.Replacing visual features with textual coordinates causes a smaller CDL decrease, while removing spatial cues causes DIOR-RSVG to fall from 77.3 to 33.5.
- Oracle and Proposer Recall Analysis: On LEVIR-MCI, proposer recall reaches 87.1% but standard F1 is 58.8% versus 70.0% oracle, making selection the dominant bottleneck.For S2Looking, both proposer recall and the standard-to-oracle gap indicate limitations on the proposal and selection sides.
- Target-Density Analysis: As target density increases, TEOChat falls from 47.6 to 26.2 F1, whereas the proposed model remains at 58.1 in the densest LEVIR-MCI bin.Degenerate arithmetic coordinate patterns occur in 19.0%–28.8% of Qwen3-VL outputs and 9.7% of Ovis2.5-FT outputs, but cannot arise from discrete region selection.
5 Conclusion
The paper presents region selection for remote sensing localization and multi-temporal change localization, replacing coordinate generation with discrete region-token selection. It reports gains on localization and grounding while retaining competitive understanding and enabling proposer–selector diagnostics.
- The framework targets both remote sensing localization and multi-temporal change localization.
- Region selection encodes candidate regions with per-frame visual features, spatial geometry, and temporal cues for discrete-token localization.
- Region selection yields substantial improvements on change detection localization and visual grounding tasks.
- Understanding tasks remain competitive, indicating that the region interface does not degrade general capabilities.
- Oracle analysis decomposes proposer and selector contributions and provides concrete guidance for future improvement.
Limitations
The framework is constrained by candidate coverage, a mismatch between token-level training and set-level evaluation, and its current axis-aligned box and short-sequence scope.
- Regions absent from the proposal set cannot be recovered, creating a proposer-recall ceiling.The analysis identifies 9.8- and 11.2-point gaps on S2Looking and LEVIR-MCI, respectively.
- Token-level cross-entropy training mismatches set-level F1 evaluation at a fixed IoU threshold.Correct region sets in a different order still incur training loss.
- The current framework operates at the axis-aligned bounding-box level and requires sufficient training data for oriented boxes or pixel masks.
- Temporal diagnostics support aligned candidate-feature comparison rather than broader reasoning over long sequences.
- The region proposal module uses a text-conditioned detector with class filtering, confidence filtering, NMS, and a 100-region cap.Performance stabilizes beyond 50 candidates, with at most a 1.1-point variation across tested caps.
B Training Details
Training starts from Ovis2.5 and proceeds through staged region-module alignment before joint multi-task adaptation with LoRA and region components.
- Ovis2.5 provides the base multimodal model, combining a NaViT vision transformer with a Qwen3 language backbone.
- Region features come from NaViT layers {6, 13, 19, 26} with 2×2 ROIAlign spatial bins.
- Training uses bf16 precision, gradient checkpointing, and multi-GPU DeepSpeed ZeRO optimization.
- Stage 2 trains region components while keeping the LLM frozen, then expands to multi-temporal and multi-target tasks.
- Stage 3 jointly trains the LLM through LoRA and all region components on the full multi-task mixture.
- The region proposal module uses a 36-category union class set.
D Dataset Summary
The dataset summary is provided in Table 9.
- Table 9 summarizes the training and evaluation data.
E Comparison with Recent Generalist MLLMs
The paper evaluates two recent open-weight generalist MLLMs under the same protocol and output adapters across five localization benchmarks, alongside the models from Table 2.
- Qwen3.5-9B and InternVL3.5-8B are evaluated zero-shot on five localization benchmarks.The comparison uses the same evaluation protocol and output adapters as Section 4.1.
F Full Component Ablation
The ablation separates spatial and temporal cues and finds that spatial information contributes most of the observed performance.
- Removing the spatial cue causes most degradation, with DIOR-RSVG falling from 77.3 to 33.5.
- Removing only the temporal cue produces small but consistent drops across all three evaluated benchmarks.
G Inference Efficiency
Region selection reduces generated output length and total inference time relative to coordinate generation, while the model qualitatively selects queried regions and temporal frames across diverse tasks.
- Inference efficiency: Region selection shortens output from 39.1 to 6.4 tokens per sample on S2Looking and from 26.0 to 3.4 on DIOR-RSVG.
- Inference efficiency: The region proposal module adds 0.11 to 0.22 seconds per sample, while injected region tokens add prefill rather than decoding cost.
- Inference efficiency: Total wall-clock time drops by a factor of 4.7 to 5.3 under identical hardware and batch settings.
- Qualitative evaluation: Qualitative examples show correct region selections across spatial referring, visual grounding, temporal referring, and change-detection tasks.