EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues

1. Introduction

• EarthDial covers RGB, SAR, multi-temporal, high-resolution satellite and aerial imagery at varying resolutions, and performs classification, detection, captioning, VQA, and grounding.Source: Figure 1 · p. 1, §1 · p. 2
• Generic VLMs do not scale well to EO data’s geospatial, spectral, and temporal dimensions; even GPT-4V shows low accuracy on domain-specific remote-sensing data [68].Source: §1 · p. 1
• Prior domain-specific VLM efforts remain limited in high-resolution processing and do not support multi-spectral or multi-temporal analysis.Source: §1 · p. 2
• The instruction-tuning dataset contains over 11.11M pairs spanning visible imagery, SAR, NIR, and infrared modalities.Source: §1 · p. 2
• EarthDial achieves higher accuracies and better generalization across 44 downstream EO tasks than existing domain-specific VLMs.The paper presents EarthDial as a unified conversational model for multi-resolution, multi-spectral, and multi-temporal remote-sensing imagery.Source: §1 · p. 2

2. Related Work

• Generic VLMs struggle with remote sensing’s geospatial, spectral, and temporal complexity, while existing geospatial VLMs lack multi-spectral and temporal coverage and varying-resolution generalization.Source: §2 · p. 2
• EarthDial’s architecture converts multi-resolution inputs into local-and-global tokens, fuses multi-channel features, and passes visual tokens with text to an LLM for tasks including VQA, detection, grounding, and change detection.Source: Figure 2 · p. 3
• Existing instruction datasets cover optical data or selected optical, SAR, and infrared imagery, but lack a unified, large-scale collection spanning diverse applications, resolutions, spectral bands, and time series.Source: §2 · p. 3

3. EarthDial

• EarthDial combines over 11M remote-sensing instructions with a unified model for multi-resolution, multi-spectral, and multi-temporal imagery and tasks from classification to change detection.Source: §3 · p. 3
• The model builds on InternVL and uses a three-stage fine-tuning recipe to progressively expand capabilities across remote-sensing modalities.Stage 1 adapts visual features; later stages use RGB and temporal imagery, then multispectral and SAR datasets.Source: §3 · p. 3

3.1. Model architecture

• A 4B-parameter model connects InternViT-300M to Phi-3-mini through an MLP projector.Source: §3.1 · p. 4
• Adaptive high resolution tiles images into 448×448 patches and adds a thumbnail for global context, using up to 12 tiles in training and 40 in inference.Source: §3.1 · p. 4
• The data-fusion module processes multispectral inputs three channels at a time, aggregates their features, and reduces token size through bilinear interpolation.Source: §3.1 · p. 4

3.2. Stage-wise Model Training

• Stage 1 pretraining aligns remote-sensing imagery with natural-language descriptions using the vision encoder, MLP, and LLM, without data fusion or temporal and multispectral datasets, beginning with RGB EarthDial-Instruct images.The Satlas and Skyscript datasets contain 7.6M image-text pairs for different types of remote-sensing imagery.Source: §3.2 · p. 4
• Stage 2 fine-tunes the MLP and LLM on RGB and temporal tasks, including visual question answering and change detection.Source: §3.2 · p. 5
• Stage 3 adds data fusion for multispectral, high-resolution RGBI, and SAR tasks while keeping the ViT parameters frozen from Stage 1.Source: §3.2 · p. 5

4. EarthDial-Instruct Dataset

• Pretraining QA pairs use SkyScript and SatlasPretrain imagery; curation filters sparse labels, cloudy images, and low spatial coverage before generating label-based instructions.Sources include Sentinel-1, Sentinel-2, NAIP, and Landsat imagery.Source: §4 · p. 5
• Table 2 summarizes QA instruction-pair counts by stage, image sources, and token formats, while Table 3 lists stages, hyperparameters, and training duration.Source: Table 2 · p. 5, Table 3 · p. 5
• Table 5 reports average classification-accuracy gains over GPT-4o of 32.5% for multispectral data and 40.2% for RGBI, plus higher SAR mAP@0.5.Table 4 compares classification accuracy across datasets and reports significant improvement over generic and specialized VLMs.Source: Table 5 · p. 6, Table 4 · p. 6
• The instructions cover ten downstream tasks, six visual modalities, and optical and SAR temporal inputs, including scene classification, detection, change detection, methane, urban heat, and disaster assessment.The tasks also include VQA, image captioning, tree-species classification, and local climate zones.Source: §4 · p. 6

5. Experiments

• EarthDial consistently outperforms compared VLMs across referred object detection, region captioning, and grounding-description benchmarks.The authors describe it as a clear winner, particularly on grounding-description tasks; the tables cover detection across datasets, object sizes, and counts.Source: §5 · p. 6, Table 6 · p. 7, Table 7 · p. 7
• EarthDial outperforms existing VLMs on image-captioning datasets and most evaluated VQA categories, while Table 11 reports its change-detection comparison.The VQA evaluation uses RSVQA-LRBEN and RSVQA-HRBEN; the supplied passages do not give the table's individual scores.Source: §5 · p. 7, Table 9 · p. 7, Table 10 · p. 7
• EarthDial reportedly improves performance across xBD temporal tasks; on QuakeSet SAR earthquake classification, its accuracy is 57.53 versus GPT-4o's 55.86.The xBD evaluation spans image captioning, region and image classification, object detection, and referred object detection; Table 12 summarizes task results.Source: §5 · p. 8, Table 12 · p. 8
• EarthDial outperforms GPT-4o by 32.16% on methane-plume accuracy, reaching 77.09% versus 40.93%, and reaches 56.77% Landsat8 UHI accuracy versus 22.68%.The methane task uses RGB and mag1c bands; the UHI task classifies regions as cooler, mildly hot, or extremely hot.Source: §5 · p. 8

6. Ablation Study

• 5% higher mAP @0.5 for referred-object detection with multi-stage pretraining than without pretraining.The table caption reports a 9.5% improvement in average classification accuracy from bilinear fusion.Source: §6 · p. 8, Table 13 · p. 9
• Bilinear interpolation fusion improves average classification accuracy over simple average pooling.Table 13 reports a 9.5% improvement for bilinear fusion.Source: §6 · p. 8, Table 13 · p. 9
• 1.75% higher classification accuracy on multispectral BigEarthNet than on its RGB counterpart.The ablation also reports bilinear interpolation fusion as more effective than average pooling; the prose gives 13.5% while Table 13 gives 9.5% for average classification accuracy.Source: §6 · pp. 8–9, Table 13 · p. 9

7. Conclusion

• EarthDial supports multi-spectral, multi-temporal, and multi-resolution imagery for classification, detection, captioning, question answering, visual reasoning, and visual grounding.Its instruction-tuning dataset contains over 11M pairs covering RGB, Sentinel-2, SAR, NIR, and infrared.Source: §7 · pp. 9–10
• Across 44 downstream tasks, EarthDial outperforms existing generic and domain-specific models, with reported generalization across EO tasks.The conclusion also highlights bi-temporal and multi-temporal sequence analysis for change detection and disaster assessment.Source: §7 · p. 10

Appendix

• The appendix details the EarthDial-Instruct dataset used to train the model.It describes the dataset in three stages.Source: Appendix · p. 10
• An ablation compares LoRA fine-tuning with full fine-tuning on zero-shot detection datasets.Source: Appendix · p. 10
• Qualitative comparisons with recent VLMs examine generalization across modalities, resolutions, and temporal EO tasks.Source: Appendix · p. 10

A. EarthDial-Instruct Dataset

• Samples from SkyScript and SatlasPretrain pair Sentinel-2, Sentinel-1 SAR, NAIP, and Landsat imagery with labels.The pipeline selects samples with at least three labels, then filters cloudy images and those with insufficient spatial coverage using luminance and coverage criteria.Source: A. EarthDial-Instruct Dataset · p. 10, Figure A.1 · p. 10
• InternLM-XComposer2 generates QA pairs from formatted label attributes, including points, polygons, object categories, and positions.The pipeline uses OSM-associated labels and imagery, and repeats generation up to five times if the expected format is not respected.Source: A. EarthDial-Instruct Dataset · p. 10, Figure A.1 · p. 10
• Prompts request objective answers grounded in visible image elements and ask about an attribute's position, interactions, characteristics, or function.Each prompt includes a satisfactory QA example and information about the other detected attributes; randomly drawn instruction sets are manually checked.Source: A. EarthDial-Instruct Dataset · pp. 10–11

• The curated instruction-following datasets cover ten downstream tasks, including classification, detection, question answering, captioning, change detection, and disaster assessment.Source: Downstream Tasks Image-text Instruction · p. 11
• The visual data span Optical, SAR, S2, Infrared, NIR, Landsat8, and Hyperspectral modalities.Source: Downstream Tasks Image-text Instruction · p. 11
• Temporal inputs cover Optical and SAR modalities.Source: Downstream Tasks Image-text Instruction · p. 11

B. Ablation on LoRA vs Full Fine-tuning

• LoRA retains Stage 1 pretrained weights and trains only low-rank adapter weights, which are added to the original weights.Source: B. Ablation on LoRA vs Full Fine-tuning · p. 11
• The LoRA setup updated approximately 201M of the model’s 4 billion parameters, with the ViT, MLP, and LLM components frozen.Source: B. Ablation on LoRA vs Full Fine-tuning · p. 11
• LoRA underperforms the fully fine-tuned model on zero-shot detection, which the authors attribute to limited parameter updates, frozen components, and constrained adaptability.Source: B. Ablation on LoRA vs Full Fine-tuning · p. 11
• 2.6 to 6.7 for multiple objects and 9.2 to 25.67 for large objects on Urban Tree Crown Detection, favoring EarthDial over EarthDial-Lora.The authors also report that EarthDial significantly outperforms EarthDial-Lora across all metrics and that the same trend appears on Swimming Pool.Source: B. Ablation on LoRA vs Full Fine-tuning · p. 11

B.1. Qualitative Analysis

• EarthDial is reported to detect objects better in crowded SAR and infrared scenes, produce multiple labels for multi-label classification, and identify semantic changes in complex temporal scenes.The comparisons include InternVL-4B [12], GPT-4o [45], and GeoChat [28]; the figure caption describes the comparison as qualitative.Source: §B.1 · p. 11, Figure A.3 · p. 14
• The qualitative discussion also reports stronger disaster understanding on optical and SAR imagery and better handling of multispectral imagery than GPT-4o on RGB+NIR and S2 inputs.InternVL-4B and GeoChat are described as not supporting multispectral processing.Source: §B.1 · p. 11
• EarthDial can fail to identify a ship bounding box; the authors point to limited ship information in training relative to vehicles and suggest more SAR ship QA pairs might help.Source: §B.1 · pp. 11, 13
• The figure illustrates EarthDial across modalities, resolutions, spectral bands, temporal data, and a range of remote-sensing tasks.Source: Figure A.2 · p. 13
• Subtle temporal changes can be missed, while similar office-building and multi-unit-residential scenes may also confuse temporal scene classification.Source: §B.1 · p. 13