Source-linked AI summary
VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis
Chao Pang, Xingxing Weng, Jiang Wu, Jiayu Li, Yi Liu, Jiaxing Sun, Weijia Li, Shuai Wang, Litong Feng, Gui-Song Xia, Conghui He
TL;DR
Remote-sensing VLM research has relied on sparse-content captions and factual-only instruction datasets, leaving these approaches limited in caption detail and deceptive-question coverage. VHM combines rich-caption and honest-instruction datasets with two-stage training and multi-level visual representations; it achieves state-of-the-art performance across multiple common remote-sensing tasks and shows potential on additional tasks.
Problem
Remote-sensing image-text datasets often describe only prominent objects, while existing instruction datasets contain factual questions, leaving these data resources limited in caption detail and deceptive-question coverage.
Source: §1 · p. 1, §2.2 · p. 3Method
VHM combines rich-content captions in VersaD and factual and deceptive questions in HnstD with two-stage training and multi-level visual representations.
Source: §1 · p. 2, §2.2 · p. 3Results
Takeaways & Limitations
The paper reports potential for VLMs to facilitate more remote-sensing tasks and notes that model honesty is crucial in applications such as national defense security.
Source: §5 · p. 7
Abstract
from arXiv · showhide
This paper develops a Versatile and Honest vision language Model (VHM) for remote sensing image analysis. VHM is built on a large-scale remote sensing image-text dataset with rich-content captions (VersaD), and an honest instruction dataset comprising both factual and deceptive questions (HnstD). Unlike prevailing remote sensing image-text datasets, in which image captions focus on a few prominent objects and their relationships, VersaD captions provide detailed information about image properties, object attributes, and the overall scene. This comprehensive captioning enables VHM to thoroughly understand remote sensing images and perform diverse remote sensing tasks. Moreover, different from existing remote sensing instruction datasets that only include factual questions, HnstD contains additional deceptive questions stemming from the non-existence of objects. This feature prevents VHM from producing affirmative answers to nonsense queries, thereby ensuring its honesty. In our experiments, VHM significantly outperforms various vision language models on common tasks of scene classification, visual question answering, and visual grounding. Additionally, VHM achieves competent performance on several unexplored tasks, such as building vectorizing, multi-label classification and honest question answering. We will release the code, data and model weights at https://github.com/opendatalab/VHM .
1 Introduction
Remote-sensing VLMs face two limitations: sparse-content captions constrain image understanding, while factual-only instruction data can lead to affirmative answers about nonexistent objects. VHM addresses these gaps with VersaD, HnstD, and a two-stage model that performs across common and additional remote-sensing tasks.
1 Introduction
- Existing remote-sensing image-text pairs often describe only prominent objects and their relationships, omitting attributes such as color and shape.The paper says this limits thorough image understanding and diverse remote-sensing analysis.Source: §1 · p. 1
- Factual-only instruction datasets can make VLMs answer deceptive questions affirmatively about nonexistent objects.The paper gives misrepresenting the color of a nonexistent object as an example.Source: §1 · pp. 1–2
- VersaD uses rich-content captions describing image properties, object attributes, and scene context to support thorough image understanding.The authors prompt off-the-shelf models to include metadata, attributes, and scene context, with constraints on uncertain object descriptions.Source: §1 · p. 2
- HnstD pairs images with factual and deceptive questions about object presence, attributes, and relative positions, and is used for instruction fine-tuning.The authors also introduce honest question answering to compare model honesty.Source: §1 · p. 2
- VHM combines the datasets with two-stage training and multi-level visual representations, performing more downstream tasks and achieving state-of-the-art results across multiple common remote-sensing tasks.The paper names building vectorizing and multi-label classification among the additional tasks.Source: §1 · p. 2
2 Versatile and Honest Datasets
This section introduces VersaD and HnstD as complementary datasets for remote-sensing image analysis and honest question answering. It describes their construction, quality assessment, task coverage, and evaluation.
2.1 VersaD
- VersaD draws nearly 1.4 million images from diverse open-source datasets, spanning 0.08–153 meters per pixel, and uses Gemini-Vision prompts for image properties, object attributes, scene context, and uncertainty.The prompts also constrain answer format to improve caption accuracy.Source: §2.1 · pp. 2–3
- 73% of sampled caption sentences were completely accurate, 10% completely inaccurate, and 17% partially accurate.Among partially accurate sentences, 55% of information pieces were accurate.Source: §2.1 · p. 3
- 82.3% overall accuracy for VersaD surpasses CC3M’s 79%, and the paper reports that VLMs pretrained on VersaD outperform those trained on sparse-content, accurate-caption data.The authors infer that rich caption content can compensate for noise.Source: §2.1 · p. 3
- VersaD-Instruct contains 30K images, with 26K for conversation and 4K for complex reasoning, using rich captions and bounding-box annotations to generate data.The images come from DOTA-v2, Fair1M, and DIOR.Source: §2.1 · p. 3
2.2 HnstD
- HnstD contains over 45K image-question-answer pairs across presence, color, relative-position, and absolute-position tasks; all but presence include factual and deceptive questions.Questions and answers are manually checked; formats include yes/no presence questions, open-ended color questions, and five-choice position questions.Source: §2.2 · pp. 3–4
- Honest question answering scores presence, relative-position, and absolute-position tasks by matching, averages factual and deceptive accuracy for the latter two, and evaluates color with matching plus ChatGPT-3.5.Color deception includes questions about nonexistent objects and objects in panchromatic images.Source: §2.2 · p. 4
3 Versatile and Honest VLM
VHM combines an image encoder, projection layer, and language model, using multi-level visual features to represent remote-sensing images. It is trained first on VersaD and then fine-tuned on three instruction datasets for dialogue, task-specific analysis, and honesty.
3.1 Model Architecture
- The vision encoder compresses remote-sensing images, the projection layer bridges images and language, and the LLM reasons over visual and textual information.VHM uses CLIP-Large at 336×336 resolution and concatenates image tokens from Transformer layers 8, 16, and 24 before compression and projection.Source: §3.1 · p. 4
- VHM concatenates features from the 8th, 16th, and 24th Transformer layers to preserve spatial information useful for object localization.The resulting visual tokens pass through compression and projection layers before reaching the language model.Source: §3.1 · p. 4
3.2 Training Strategy
- Training starts from LLaVA weights, pretrains all VHM components on VersaD, then fine-tunes the compression layer, projection layer, and LLM on three instruction datasets.The fine-tuning datasets are VersaD-Instruct, VariousRS-Instruct, and HnstD; only the vision encoder is frozen.Source: §3.2 · p. 5
4 Experiments
Across common remote-sensing tasks, VHM reports gains in scene classification, visual question answering, and visual grounding, while also evaluating specialized capabilities and honest question answering. Experiments include dataset comparisons, qualitative examples, and ablations of pretraining data and visual representations.
4.1 Datasets
- The comparison spans five scene-classification datasets, two visual-question-answering datasets, and one visual-grounding dataset, with additional tests for VHM-specific capabilities.The listed datasets include NWPU, METER-ML, SIRI-WHU, AID, WHU-RS19, RSVQA-LR, RSVQA-HR, and DIOR-RSVG.Source: §4.1 · p. 5
4.2 Evaluation on Versatility
- VHM demonstrates image-attribute recognition, building vectorizing, and zero-shot multi-label classification, but struggles with object counting and geometric measurement.It reaches 95% accuracy on image modality and 0.24 mean absolute error on image resolution; counting and geometric-measurement errors are 6.75 and 12.82.Source: §4.2 · p. 5, Table 5 · p. 5
- 10.6% and 8.22% improvements on NWPU and SIRI-WHU accompany zero-shot VQA performance 2.6% above EarthGPT; supervised VQA is comparable to LHRS-Bot (89.33% vs 89.19%).The VQA comparison uses the same training setting of one epoch and 10K samples for VHM and LHRS-Bot.Source: §4.2 · p. 6, Table 6 · p. 6, Table 7 · p. 6
- VHM surpasses the best visual-grounding competitor by 11.59, even at 336×336 input size, using spatial information in low-level features.DIOR-RSVG images are 800×800 originally; downsampling reduces object features and makes localization harder.Source: §4.2 · p. 6, Table 9 · p. 6
- Qualitative examples show VHM describing objects, scenes, attributes, and layout while answering counting, relative-position, and resolution questions.The examples also illustrate the paper’s claim that VHM provides honest answers.Source: §4.2 · p. 6, Figure 5 · p. 6
4.3 Evaluation on Honesty
- On HnstD, VHM exceeds competitors by 10.35% on factual presence questions and performs well across factual and deceptive questions, especially for color and absolute position.VHM’s relative-position accuracy is described as remarkable, though the authors note room for improvement.Source: §4.3 · pp. 6–7, Table 11 · p. 7
4.4 Ablation Studies
- RS pretraining beats the LLaVA-initialized baseline across tasks, while VersaD yields over 43% visual-grounding improvement over SkyScript despite similar pair counts.The authors infer that rich-content captions also make VLMs less sensitive to label noise.Source: §4.4 · p. 7, Table 10 · p. 7
- Multi-level visual representations improve object-localization accuracy by over 9% compared with single-level features.The authors attribute the gain to detailed spatial information in low-level image features.Source: §4.4 · p. 7, Table 12 · p. 7
5 Conclusion
VHM combines rich-content remote-sensing captions with factual and deceptive instruction questions, and uses two-stage training with multi-level visual representations. It lacks pixel-wise perception, limiting its ability to perform semantic segmentation or change detection.
5 Conclusion
- VHM combines VersaD’s rich-content captions and HnstD’s factual and deceptive questions with two-stage training and multi-level visual representations.The paper reports state-of-the-art performance across public datasets and multiple common remote-sensing tasks, and potential on additional tasks.Source: §5 · p. 7
- VHM currently lacks pixel-wise perception, preventing semantic segmentation and change detection of remote-sensing images.The paper identifies addressing this limitation as a key direction for future research.Source: §5 · p. 7
A Appendix Overview
- The supplementary material provides VersaD and HnstD details, VariousRS-Instruct construction, experiment results, and related work.It includes dataset and metric details and additional qualitative results.Source: A Appendix Overview · p. 10
B.1 Samples in VersaD
These supplementary sections show VersaD caption examples, assess their accuracy, and describe how rich captions and bounding boxes support VersaD-Instruct conversation generation.
B.1 Samples in VersaD
- VersaD captions convey image properties, object attributes, and the overall scene, as illustrated by dataset samples.The captions are generated using proposed prompts and Gemini-Vision.Source: §B.1 · p. 10, Figure 6 · p. 10
B.2 VersaD Quality Assessment
- 82.3% is the reported accuracy of VersaD, estimated from sentence-level judgments of 315 randomly sampled image-text pairs.The sample yielded 2,002 sentences; 73% were completely accurate, 10% completely inaccurate, and 17% partially accurate, with 55% of information pieces accurate in the partial sentences.Source: §B.2 · p. 10
B.3 VersaD-Instruct Construction
- Language-only Gemini uses rich-content captions and bounding-box annotations to generate VersaD-Instruct multi-turn conversations and complex reasoning.The conversations include information such as object colors and quantities.Source: §B.3 · p. 10
C.1 Task Construction of HnstD
HnstD covers four recognition tasks and constructs factual and deceptive questions using image content and bounding-box annotations.
C.1 Task Construction of HnstD
- HnstD comprises relative position, presence, color, and absolute position recognition tasks, identified with {IDK}.The section points to task prompts and templates in Figures 10 and 11.Source: §C.1 · p. 10
- Presence questions use bounding boxes for “Yes” examples and random, popular, or adversarial strategies to select absent objects for “No” examples.The adversarial strategy selects absent objects that frequently co-occur with objects present in the image.Source: §C.1 · pp. 10–11
- Color questions use Gemini-Vision on enlarged single-object crops, accepting extracted color only when two prompted answers agree.Deceptive color questions include absent objects and objects in panchromatic DOTA-v2 images; factual questions use color and bounding-box information.Source: §C.1 · p. 11
- Absolute-position questions use single-object images divided into nine regions, with deceptive examples generated by selecting absent objects.Position is determined using established rules, such as left side, top right corner, or center.Source: §C.1 · p. 11
- Relative-position questions use images with two object categories and one instance per category, deriving positions from their bounding boxes.Deceptive examples select one or two absent objects using three strategies.Source: §C.1 · p. 11
C.2 Performance Evaluation with ChatGPT-3.5
For color-task performance evaluation, deceptive questions have open-ended answers and are assessed with ChatGPT-3.5.
C.2 Performance Evaluation with ChatGPT-3.5
- ChatGPT-3.5 is used to evaluate performance on the color task’s deceptive questions.The section identifies Figure 13 as containing the evaluation prompts.Source: §C.2 · p. 11
- Deceptive questions in the color task are paired with open-ended answers.The passage specifies this answer format for evaluation with ChatGPT-3.5.Source: §C.2 · p. 11
- Figure 13 presents the prompts used for ChatGPT-3.5 performance evaluation.Source: §C.2 · p. 11
D More Details about VariousRS-Instruct
VariousRS-Instruct broadens remote-sensing instruction beyond common VQA and grounding tasks to include object counting, image-property recognition, geometric measurement, building vectorizing, and multi-label classification.
D More Details about VariousRS-Instruct
- Its in-context example pairs a detailed aerial-image caption and object coordinates with multi-turn questions and answers about tennis courts.The example includes counting, material, spacing, fencing, and a qualified inference about whether the courts might host large-scale events.Source: D More Details about VariousRS-Instruct · p. 12
- The appendix provides prompts and templates for constructing VariousRS-Instruct, plus language-only Gemini in-context examples for creating VersaD-Instruct.Figure 9 is identified as in-context examples; Figures 14 and 15 are identified as prompts and templates.Source: Figure 9 · p. 12, D More Details about VariousRS-Instruct · p. 12
- VariousRS-Instruct covers object counting, image-property recognition, geometric measurement, building vectorizing, and multi-label classification alongside VQA, visual grounding, and scene classification.The paper assigns {VQA}, {VG}, and {CLS} identifiers to those task types and uses {IT} for the others.Source: D More Details about VariousRS-Instruct · p. 12
E.1 Datasets and Evaluation Metrics
The evaluation uses multiple remote-sensing datasets and reports task-specific test sets and metrics; the appendix also identifies prompts and templates used in dataset construction.
E.1 Datasets and Evaluation Metrics
- VHM is evaluated on multiple remote-sensing datasets, with a test set and metrics specified for each task.Table 13 defines MAE, ACC, C-IoU, and example-based F1-measure.Source: §E.1 · p. 12, Table 13 · p. 13
- Appendix figures identify prompts and templates for HnstD construction, object-color extraction, color-task evaluation, and VariousRS-Instruct construction.Figures 10–15 are labeled as prompts or templates for these steps.Source: Figure 10 · p. 12, Figure 11 · p. 12, Figure 12 · p. 13
- Table 13 names MAE, ACC, C-IoU, and example-based F1-measure as evaluation metrics and expands each abbreviation.The table identifies them as Mean Absolute Error, Accuracy, Complexity-aware IoU, and example-based F1-measure.Source: Table 13 · p. 13
E.2 More Qualitative Results
Figure 17 presents qualitative VHM results across various remote-sensing image-analysis tasks.
E.2 More Qualitative Results
- Figure 17 presents qualitative results of VHM across various remote-sensing image-analysis tasks.The passage characterizes the examples as demonstrating the model’s versatility, without specifying individual outcomes.Source: §E.2 · p. 13
F Related Works
Prior remote-sensing vision-language models cover common tasks but have reported gaps in geometric understanding, building vectorization, and answering deceptive queries. This paper positions VHM’s datasets and broader task coverage as a response to those gaps.
F Related Works
- Existing remote-sensing VLMs commonly support VQA, visual grounding, and scene classification, while some also support referring expressions or object detection.LHRS-Bot extends task coverage to image-attribute recognition and object-relationship awareness, but the passage says geometric information and building vectorization remain challenging.Source: F Related Works · p. 13
- Existing RS image-text datasets are described as having sparse captions that focus on a few objects and relationships while omitting details such as color and shape.The passage says this limits thorough image understanding and the ability to accomplish more tasks.Source: F Related Works · p. 14
- Existing RS instruction datasets are described as containing only factual questions, leaving VLMs susceptible to affirmative answers to nonsense queries.The paper presents rich-caption image-text data and an instruction dataset with factual and deceptive questions as its response to these gaps.Source: F Related Works · p. 14