PaperlayerView exact revision on arXiv

Source-linked AI summary

UniGeo: A Multi-modal Large Language Model for Text-Guided Cross-View Geo-Localization

Jiahao Wen, Hang Yu, Zhedong Zheng

arXiv:2608.26722v1cs.CV

TL;DR

Text-guided drone geo-localization is difficult because natural-language queries can omit spatial and viewpoint cues, while candidates may be geographically incorrect despite semantic relevance. UniGeo combines candidate-conditioned query refinement, geo-semantic learning, cross-view generation, and candidate-level verification; it improves retrieval performance while preserving the original backbone.

  • Problem

    Incomplete text descriptions and geographically incorrect but semantically relevant candidates make text-query drone geo-localization challenging.

  • Method

    UniGeo uses candidate-conditioned query refinement and candidate-level verification within a unified geospatial vision-language framework, with region-aware geo-semantic learning and cross-view generation.

  • Results

    UniGeo improves R@10 by 13.6 percentage points without modifying the retrieval backbone, while improving the image-query baseline from 28.81/56.45/69.30 to 29.32/56.80/69.57 for R@1/R@5/R@10.

  • Takeaways & Limitations

    UniGeo mainly promotes correct targets initially ranked lower into the leading candidate set, supporting accuracy-critical fine-grained localization.

  • Takeaways & Limitations

    Aggressively altering the top-1 ranking can be unstable when refined queries contain uncertain or over-specified semantics.

Abstract

from arXiv · show

Text-guided drone geo-localization aims to identify a target region in a large-scale image gallery from a natural-language description. Existing methods mainly formulate this task as direct matching between an open-ended text query and candidate images. However, incomplete queries and highly similar candidates often make global cross-modal matching insufficient for reliable fine-grained localization. We propose UniGeo, a unified multimodal large language model (MLLM) for text-guided drone geo-localization. Built on a shared vision-language framework, UniGeo jointly supports geo-semantic understanding, cross-view semantic generation, and candidate-level verification. Specifically, it establishes stable correspondences among local scene elements, spatial relations, and language descriptions through geo-semantic learning, and further models semantic mappings between drone and satellite views through cross-view generation. Based on these capabilities, a plug-and-play verification module performs fine-grained discrimination among highly confusable candidates. We further introduce a multi-stage training strategy that progressively learns geo-semantic understanding, cross-view generation, and candidate verification, improving adaptation to text-guided geo-localization. Experiments demonstrate consistent improvements across multiple retrieval backbones. On GeoText-1652, UniGeo improves R@10 and mAP by 13.59 and 2.83 percentage points, respectively, validating its effectiveness for fine-grained text-guided drone geo-localization.

I. INTRODUCTION

Text-guided drone geo-localization is practically important because users describe target regions through natural language, while incomplete queries and similar candidates challenge direct matching. UniGeo addresses this with unified geo-semantic understanding, cross-view generation, and candidate verification compatible with existing retrieval backbones.

  • Natural-language descriptions make drone geo-localization practically relevant by expressing appearance, spatial layout, and relations to surrounding structures.
  • Existing methods mainly directly match open-ended queries to candidates, although queries may lack stable and discriminative geo-localization evidence.
  • UniGeo unifies region grounding, captioning, region-grounded description, image-text matching, cross-view generation, and candidate verification in one MLLM.
  • The framework targets geo-semantic understanding, cross-view semantic generation, and fine-grained verification to refine queries and distinguish confusable candidates.
  • UniGeo consistently improves Text Query mAP over several retrieval backbones while adding only marginal parameters.

II. RELATED WORK

Related work spans cross-view geo-localization, vision-language retrieval, and unified vision-language modeling. UniGeo builds on these directions while emphasizing unified understanding, generation, and reasoning for geospatial localization.

  • Cross-view geo-localization addresses correspondence between observations from different platforms despite viewpoint changes, layout distortion, and imaging differences.
  • GeoText-1652 introduced natural language for drone-to-satellite retrieval, while later methods improved alignment and transfer mechanisms.
  • Vision-language pretraining established shared image-text embedding spaces that support transferable recognition and retrieval.
  • Unified vision-language models increasingly combine retrieval, generation, and reasoning within a single framework.

III. METHOD

UniGeo defines a shared geospatial representation space and acquires its capabilities through progressive optimization of grounding, cross-view augmentation, and candidate verification.

  • UniGeo uses a unified geospatial representation space for local semantics, language-to-region correspondence, and structured spatial relations.
  • Its Progressive Learning Policy optimizes geospatial grounding, cross-view semantic augmentation, and candidate-level verification in sequence.

A. UniGeo Framework

The UniGeo framework learns local language-region correspondences and spatial organization, then supports candidate-conditioned refinement and verification within a shared MLLM.

  • UniGeo Framework: UniGeo uses a shared vision-language backbone with region-level semantic and structured discriminative supervision.
  • Region-level Semantic Supervision: Region-level supervision generates descriptions for specified local regions using bounding-box conditioning, image overlays, and optional ROI crops.
  • Region-level Semantic Supervision: The conditional language objective establishes stable correspondences between local visual regions and language descriptions.
  • UniGeo Framework: The framework combines unified tokens, task-specific heads, progressive three-stage learning, and plug-and-play candidate re-ranking.
  • Structured Discriminative Supervision: A language-to-region head predicts referred bounding boxes from regional text and full-image tokens, explicitly enforcing grounding.
  • Structured Discriminative Supervision: Spatial supervision classifies horizontal relations as left, center, or right and vertical relations as upper, middle, or lower.

B. Progressive Learning Policy

UniGeo uses a three-stage progressive strategy to learn geo-semantic understanding, pose-aware cross-view generation, and hard-negative candidate verification.

  • Stage-I: Geo-Semantic Understanding Learning: Stage I jointly learns region semantics, image-text alignment, language-to-region grounding, and inter-region spatial structure.The objective combines region-level semantic, global alignment, fine-grained matching, spatial, and bbox-grounding supervision.
  • Stage-II: Pose-Aware Cross-View Generation: Stage II adds pose-aware latent-diffusion generation to model semantic and geometric consistency across satellite and drone viewpoints.The shared vision-language backbone is inherited and frozen, while a pose teacher supplies heading and range supervision.
  • Stage-II: Pose-Aware Cross-View Generation: The Stage-II objective combines diffusion denoising, pose consistency, and auxiliary heading regularization.Pose consistency decomposes into heading and range terms, with the heading term comparing predicted vectors against sine-cosine angle encodings.
  • Stage-III: Hard-Negative Verification Head Learning: Stage III freezes the backbone and generation branch while training a lightweight verification head on real and synthetic hard negatives.The head scores query-candidate fused representations, using binary matching losses for positive pairs and hard negatives.
  • Stage-III: Hard-Negative Verification Head Learning: Stage III integrates grounding and cross-view generation to improve candidate-level discrimination among highly confusable candidates.Synthetic negatives from the generation branch provide harder supervision than random negatives.

C. Plug-and-Play Inference

At inference, UniGeo preserves an external backbone’s high-recall retrieval and applies candidate-conditioned refinement, dual-query scoring, and local verification reranking.

  • Plug-and-Play Inference: UniGeo is plug-and-play at inference and compatible with arbitrary external retrieval backbones.It does not alter their original training protocol or initial retrieval pipeline.
  • Initial Candidate Retrieval: An external retrieval backbone first returns a top-K0 candidate pool without changing the original candidate-generation mechanism.UniGeo performs subsequent fine-grained reasoning on the retrieved candidates.
  • Candidate-Conditioned Query Refinement: Candidate-conditioned refinement supplements localization-relevant cues using the original query and initial candidate context.The refined-query candidates are unioned with the initial pool to form a shared candidate set.
  • Shared-Pool Dual-Query Scoring: UniGeo estimates refinement reliability from query consistency, candidate overlap, and newly introduced semantic content, then adaptively combines original and refined-query scores.The resulting score is the gated dual-query candidate score.
  • Local Ambiguity Correction and Geo-Localization: For a local shortlist, the verification head score is fused with the primary score before reranking; candidates outside the shortlist remain unchanged.The highest-scoring corrected candidate is selected as the geo-localization result.

A. Implementation Details

The implementation uses UniLIP with an InternVL3-2B processor, three-stage training, broad evaluation metrics, and planned public release of code and weights.

  • MLLMs: UniGeo uses UniLIP as its underlying MLLM backbone and pairs it with the InternVL3-2B processor.The framework can be integrated with arbitrary external candidate-retrieval backbones.
  • Training: Training follows a three-stage progressive strategy, beginning with region-level VQA fine-tuning and subsequent multi-task foundation training.The reported VQA setup uses one epoch, learning rate 5×10^-6, maximum sequence length 1024, batch size 1, and eight gradient accumulation steps.
  • Evaluation Metrics: Evaluation covers geo-localization, text generation, and cross-view image generation.Geo-localization uses Recall@1, Recall@5, Recall@10, and mAP; generation metrics assess text quality, image quality, and pose control.
  • Reproducibility: The authors state that code and model weights will be publicly released.

B. Main Results

UniGeo is evaluated as a plug-and-play post-retrieval reasoning framework across retrieval, query refinement, grounded description, and pose-aware cross-view generation. It consistently improves localization and candidate ranking across datasets and backbones while producing stronger grounded descriptions and cross-view images.

  • Geo-Localization Performance: UniGeo consistently improves retrieval performance across GeoText-1652 and UAVReason with pre-trained, fine-tuned, and task-specific backbones.It is applied without modifying the underlying retrieval architectures.
  • Geo-Localization Performance: UniGeo mainly improves ranking and discrimination within the leading candidate set, with larger gains at R@5 and R@10 and consistent mAP improvements.On UAVReason, UniGeo raises HCCM R@10 from 63.90 to 64.71 and another baseline from 77.33 to 77.86.
  • Image Query: Under image-query evaluation, UniGeo improves retrieval metrics on GeoText-1652 and UAVReason, indicating transfer of its candidate-level calibration across query modalities.For example, UAVReason XVLM-16M R@5/R@10 rises from 16.60/27.28 to 20.95/32.90, while mAP increases from 2.10 to 2.38.
  • Query Refinement and Grounded Description: UniGeo achieves the best reported region-grounded description quality, including Region-Match Acc@1 of 0.71 and BLEU-1/BLEU-4/F1 scores of 0.27/0.06/0.34.The results indicate stronger association between generated descriptions and target regions.
  • Pose-Aware Cross-View Generation: UniGeo produces stronger pose-aware cross-view generation, with FID/PSNR/SSIM of 41.34/13.12/0.16 in the reported reverse direction.Compared with UniLIP, FID decreases from 56.72 to 41.34 while PSNR and SSIM also improve; qualitative results show better topology, footprints, perspective, and spatial layout preservation.

C. Ablation Studies and Further Discussion

The ablations show that UniGeo's gains arise from combining query refinement, candidate verification, and hard-negative learning, while pose-aware generation balances visual fidelity with viewpoint control. The framework improves retrieval and verification across settings, though it adds inference cost.

  • Main mechanism ablation: 14.56/32.32/42.44 R@1/R@5/R@10 results show hard-negative training substantially improves text-query localization over verification without hard negatives.Hard negatives expose the verifier to highly confusable candidates and strengthen fine-grained cross-view discrimination.
  • Main mechanism ablation: 29.32/56.80/69.57 R@1/R@5/R@10 results improve the image-query baseline through hard-negative verification learning, with more moderate gains than in text-query localization.The hard-negative variant and full model are identical for image queries because query refinement is not used.
  • Hyperparameter sensitivity: 14.72/33.80/45.37 R@1/R@5/R@10 is achieved at K = 16, while larger shortlists produce negligible changes and α = 0.1 performs best.Most competitive candidates are already covered by a compact local shortlist, and nearby α values remain comparable.
  • Hard-negative supervision: The Real+Syn 2:2 hard-negative mixture achieves the highest Real Acc. and lowest Gap while retaining competitive Syn Acc.The balanced mixture helps the verification head avoid overfitting to real retrieval errors or synthetic artifacts.
  • Pose-aware generation: The full pose-aware generation objective achieves the lowest MeanIdxErr while maintaining competitive FID, PSNR, and SSIM.Diffusion reconstruction preserves image quality, pose consistency controls viewpoints, and heading supervision improves directional consistency.
  • Efficiency and backbone compatibility: UniGeo improves retrieval backbones without changing candidate generation, but requires approximately 2.6T FLOPs per query versus 106.42G for XVLM.The added overhead comes from dynamic refinement and candidate-level verification, while preserving the external retriever's high-recall process.
  • Qualitative analysis: Heading-controlled generation preserves major geo-semantic elements across viewpoints, and generated samples provide harder supervision than random negatives for verification learning.The generated images are geographically plausible and semantically confusable rather than pixel-level reconstructions.
  • Qualitative analysis: Region-level descriptions capture object categories, attributes, context, and spatial relations, supplying structured cues for subsequent geo-localization.UniGeo associates local region semantics with global aerial-scene context rather than recognizing isolated objects alone.

D. Discussion

UniGeo mainly strengthens shortlist-level recall rather than guaranteeing exact first-rank localization. Its conservative reranking strategy addresses ambiguity while avoiding unstable top-1 changes from uncertain query refinement.

  • Discussion: UniGeo's larger gains on R@5 and R@10 than R@1 reflect region-level textual cues that rarely identify a unique target.Similar roads, buildings, and spatial layouts make exact top-1 localization difficult.
  • Discussion: The framework preserves the original retrieval backbone and refines rankings among plausible candidates, promoting lower-ranked correct targets into the leading shortlist.This reduces the risk of missing correct locations during map inspection, human verification, or fine-grained localization.

V. CONCLUSION

UniGeo treats text-guided drone geo-localization as both retrieval and post-retrieval geo-semantic reasoning. Its unified pipeline combines understanding, pose-aware generation, and verification, with consistent improvements across representative retrieval backbones.

  • Conclusion: UniGeo integrates geo-semantic understanding, pose-aware cross-view generation, and candidate-level verification to mine fine-grained cues within the initial candidate pool.The pipeline addresses incomplete natural-language descriptions and visually similar geographic regions.
  • Conclusion: Experimental results consistently improve multiple representative retrieval backbones on text-query geo-localization.Additional analyses support region-level understanding, controllable generation, and hard-negative verification as parts of the framework's geospatial reasoning capability.
Loading 2608.26722v1…