Source-linked AI summary
LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model
Dilxat Muhtar, Zhenshi Li, Feng Gu, Xueliang Zhang, Pengfeng Xiao
TL;DR
Existing RS MLLMs do not adequately account for the complexity of global landscapes, varied objects, and visual scales in RS imagery. The paper builds VGI-enhanced datasets, an RS-specific MLLM, and a benchmark, finding that LHRS-Bot delivers strong RS image understanding and reasoning, while remaining susceptible to hallucinations.
Problem
Existing RS MLLMs do not adequately account for diverse geographical landscapes, varied objects, and visual-scale variability in RS imagery.
Method
The paper constructs LHRS-Align and LHRS-Instruct from worldwide RS images and OSM VGI, then develops LHRS-Bot using multi-level vision-language alignment and curriculum learning, alongside LHRS-Bench.
Results
LHRS-Bot demonstrates exceptional RS domain knowledge and surpasses both general-purpose and RS-specific MLLMs across comprehensive comparison experiments.
Takeaways & Limitations
The datasets, model, and benchmark provide a systematic foundation for developing and evaluating MLLMs specialized for RS image understanding.
Takeaways & Limitations
The authors cannot confirm whether some data appeared in the LHRS-Align dataset.
Abstract
from arXiv · showhide
The revolutionary capabilities of large language models (LLMs) have paved the way for multimodal large language models (MLLMs) and fostered diverse applications across various specialized domains. In the remote sensing (RS) field, however, the diverse geographical landscapes and varied objects in RS imagery are not adequately considered in recent MLLM endeavors. To bridge this gap, we construct a large-scale RS image-text dataset, LHRS-Align, and an informative RS-specific instruction dataset, LHRS-Instruct, leveraging the extensive volunteered geographic information (VGI) and globally available RS images. Building on this foundation, we introduce LHRS-Bot, an MLLM tailored for RS image understanding through a novel multi-level vision-language alignment strategy and a curriculum learning method. Additionally, we introduce LHRS-Bench, a benchmark for thoroughly evaluating MLLMs' abilities in RS image understanding. Comprehensive experiments demonstrate that LHRS-Bot exhibits a profound understanding of RS images and the ability to perform nuanced reasoning within the RS domain.
1 Introduction
Remote sensing image understanding remains challenging because worldwide landscapes, objects, and visual scales vary substantially, while existing RS-specific MLLMs insufficiently exploit available datasets. LHRS-Bot addresses this gap with VGI-enhanced datasets, multi-level alignment, curriculum learning, and a dedicated benchmark.
- Remote sensing image understanding is important for monitoring and analyzing Earth’s surface and environment.
- RS images challenge holistic understanding through complex objects across worldwide landscapes and variable visual scales.
- LHRS-Align contains 1.15 million high-quality RS image-text pairs created by pairing RS images with attributed OpenStreetMap information and cleaning the data.
- LHRS-Bot combines a multi-level visual representation bridging strategy with curriculum learning for RS image understanding.
- LHRS-Bot achieves state-of-the-art performance across varied RS image understanding tasks and supports intricate object detection, conversation, and visual insight generation.
- LHRS-Bench provides systematic RS-specific evaluation through 690 single-choice questions spanning five top-level dimensions and 11 fine-grained categories.
2 Dataset
The paper constructs aligned RS datasets from worldwide imagery and geographic information, enriches them with instructional and reasoning data, and introduces a structured benchmark. LHRS-Bot then uses multi-level visual representations to connect RS imagery with language processing.
- LHRS-Align: LHRS-Align geo-aligns worldwide 1-meter RS images with OpenStreetMap geographic features and their semantic attributes.
- LHRS-Align: The alignment pipeline identifies geographic anchor features, acquires centered square image regions, and forms a raw pool of around 4 million image-feature pairs.
- LHRS-Align: Attribute pruning removes visually irrelevant keys, while semantic balancing reduces the raw pool to 1.15 million images.
- LHRS-Instruct: LHRS-Instruct combines public RS caption datasets with LHRS-Align to create multimodal instruction-following data.
- LHRS-Instruct: Selected LHRS-Align images receive spatially grounded GPT-4 instructions covering visual reasoning, detailed descriptions, and object locations and quantities.
- LHRS-Bench: LHRS-Bench contains 108 images and 690 human-annotated single-choice questions across five evaluation dimensions and 11 sub-dimensions.
- LHRS-Bot: LHRS-Bot uses learnable queries and a vision perceiver to summarize multi-level visual representations before concatenating them with language embeddings for LLM processing.
3 Method
LHRS-Bot combines a universal vision encoder, a vision perceiver, and a foundational language model through multi-level visual alignment and three-stage curriculum learning. The training stages progress from caption generation to multitask instruction learning and complex conversational and reasoning instruction data.
- Model Architecture: LHRS-Bot comprises a universal vision encoder, a vision perceiver, and a foundational LLM for interpreting visual and language information.
- Model Architecture: The vision encoder uses multi-level visual features, while learnable queries summarize each level through cross-attention and MLP layers before forwarding compact image features to the LLM.This design addresses the computational overhead and potential language-information overload caused by introducing extended image tokens.
- Curriculum Learning Strategy: The curriculum strategy progressively increases task difficulty to incrementally align the vision encoder with the LLM for multiple visual tasks and complex instructions.The strategy progressively unlocks parameters and introduces increasingly complex data across stages.
- Curriculum Learning Strategy: Stage 1 trains the vision perceiver on LHRS-Align image-caption pairs to maximize the conditional probability of generating caption tokens from visual embeddings.The visual and caption embeddings are concatenated before being fed into the LLM.
- Curriculum Learning Strategy: Stage 2 fine-tunes the vision perceiver and LLM with LoRA on instruction-formatted classification, VQA, and visual-grounding datasets.The datasets are collected from public RS resources and transformed using manually designed templates.
- Curriculum Learning Strategy: Stage 3 uses LHRS-Instruct, 20K randomly selected LLaVA complex-reasoning examples, and lower-ratio multitask data to enhance conversation and reasoning.
4 Experiment
Experiments evaluate LHRS-Bot across remote-sensing classification, VQA, visual grounding, and the LHRS-Bench benchmark, alongside qualitative demonstrations and ablations. LHRS-Bot shows strong multi-task performance, while ablations support the value of domain-specific alignment data and multi-level visual representations.
- Experimental Setup: LHRS-Bot is evaluated on image classification, VQA, visual grounding, and the LHRS-Bench benchmark using diverse datasets and competing MLLMs.The evaluation includes seven classification datasets, two VQA subsets, two visual grounding datasets, and open- and closed-source baselines.
- Quantitative Evaluation: 27.51% higher average accuracy than the second-ranked method is reported across seven classification datasets.LHRS-Bot outperforms all competing models on the seven classification datasets.
- Quantitative Evaluation: LHRS-Bot achieves performance on par with leading RS-specific models on VQA and sets a new record on RSVQA-HR.The comparison notes comparable results to RSGPT despite RSGPT’s five-epoch RSVQA fine-tuning.
- Quantitative Evaluation: LHRS-Bot outperforms other visual-grounding counterparts, despite SkyEyeGPT using a more extensive object-detection training dataset.The authors associate this result with acquiring fine-grained visual information during training.
- LHRS-Bench Evaluation: On LHRS-Bench, LHRS-Bot outperforms all other open-source models, exceeds Claude, and matches GPT-4V, while substantial room for improvement remains.Its strongest relative performance is reported for spatial resolution and image modality, which were not explicitly trained for.
- Qualitative Evaluation: Qualitative examples show holistic and fine-grained image understanding, object orientation, position, quantity, visual inference, and uncertainty acknowledgment.These demonstrations are presented in Fig. 5.
- Ablation Analysis: LHRS-Align outperforms alternative alignment datasets despite containing 1.15 million images, supporting domain-specific collection, enriched captions, and data cleaning.Both LHRS-Align and SkyScript use RS images with OSM data, while LHRS-Align performs better than SkyScript.
- Ablation Analysis: Multi-level visual representation summarization performs best overall, whereas singular representations favor classification and VQA but fall short on visual grounding.The ablation compares representation strategies and query allocations across tasks.
5 Conclusion
The paper presents LHRS-Bot as an RS-domain MLLM supported by alignment and instruction datasets plus a systematic benchmark. Experiments report strong RS knowledge, while hallucinations remain a limitation shared with LLMs.
- LHRS-Bot combines LHRS-Align, LHRS-Instruct, and LHRS-Bench into a framework for RS image understanding and evaluation.
- The framework extends MLLMs toward specialized RS image understanding through domain-specific alignment and instruction-following data.
- LHRS-Bot demonstrates exceptional RS domain knowledge and surpasses both general-purpose and RS-specific MLLMs in comprehensive comparison experiments.
- Prior RS MLLMs struggled to exploit extensive global RS imagery, whereas LHRS-Bot leverages geographical information from worldwide images.
A.3 Large-Scale Remote Sensing Vision-Language Datasets
Large-scale RS image-text datasets provide substantial training material, but their geographic descriptions and caption quality remain uneven. LHRS-related dataset construction addresses these limitations through geographically grounded information and refined semantics.
- RS5M contains 5 million RS images and was the first large-scale RS image-text paired dataset.
- RS5M provides insufficient geographical descriptions of scenes and objects and lacks orthorectified satellite images.
- SkyScript uses open-source geographical data, but rule-generated captions were not thoroughly refined through a rigorous cleaning pipeline.
- The paper develops a large-scale RS vision-language alignment dataset to provide richer, higher-quality visual semantics.
B.1 Data Source
The data source covers cities worldwide using OSM geographic features and corresponding RS imagery. Filtering combines automated criteria, expert review, deduplication, pruning, balancing, and caption generation.
- The dataset covers 9,259 cities across 129 countries, while the current version uses only polygon OSM features because of RS spatial-resolution limits.
- Qualified OSM anchors are selected when their areas exceed a 128×128 pixel image and their aspect ratios are below 4.
- Automatic filtering removes keys containing name or addr, characterless values, or fewer than three distinct values before manual review.
- Expert consensus reduces the OSM key set from 10,244 to 186, with three experts spending around 2 hours each on this sole human-discrimination step.
- Duplicate key-value pairs are removed, overly frequent pairs above t = 20k are filtered, and independent sampling supplements the remainder to improve balance.
- Captions are generated with Vicuna-v1.5-13B using temperature 0.7 and top_p 0.95.
C Details about LHRS-Instruct Dataset
LHRS-Instruct combines filtered public RS data, manually designed templates, and generated visual-reasoning material. Its construction includes image and caption filtering before conversion into task-specific instruction data.
- Public caption datasets undergo image deduplication and removal of captions shorter than 10 tokens.
- CLIP-L/14 filters image-caption samples whose average similarity score is below 15%.
- Instruction-following data are generated with Vicuna-v1.5-13B using the same sampling setting described elsewhere in the pipeline.
- The dataset includes visual-reasoning instruction data with image descriptions and conversations.
- A multi-task instruction dataset is assembled from public RS training sets using manual templates for classification and visual grounding.
- Classification and visual-grounding examples are converted into instruction data by randomly selecting among task-specific templates.
E Details about Test Datasets
Test-dataset information for each task is provided in Table 9.
- Table 9 details the dataset information for each task.
F Details about LHRS-Bench Dataset
LHRS-Bench uses hierarchical ability taxonomies and task-specific evaluation procedures, with supporting dataset, prompt, and example documentation. The training setup addresses optimization stability across three stages.
- Benchmark taxonomy: The hierarchical ability taxonomy contains 5 top-level dimensions and 11 fine-grained sub-dimensions, with question-answer pairs potentially spanning multiple sub-dimensions.
- Benchmark taxonomy: Recognition covers object or scene identity and attributes, while Quantity assesses counting with Recognition as a possible prerequisite.
- Training and evaluation: Training used eight NVIDIA V100-32G GPUs; Adan was selected for first-stage performance and stability, with DeepSpeed-Zero2 enabled throughout all three stages.
- Training and evaluation: Visual-grounding and VQA prompts use target-object descriptions and questions, while classification and LHRS-Bench evaluations use shared prompts across models.
- Supplementary documentation: The supplementary materials document multi-task instruction statistics, test datasets, hyperparameters, OSM feature keys, caption-generation examples, GPT-4 generation examples, dataset templates, evaluation prompts, and benchmark examples.
- Supplementary documentation: Figures 7–9 provide examples from LHRS-Align, LHRS-Bench, and evaluations on LHRS-Bench, respectively.