PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Model

Chenyang Liu, Keyan Chen, Rui Zhao, Zhengxia Zou, Zhenwei Shi

arXiv:2501.00895v2cs.CV

TL;DR

Remote sensing text2image generation lacks large, diverse datasets and methods for global-scale, multi-resolution, unbounded image synthesis. The paper introduces Git-10M and the diffusion-based Text2Earth foundation model, which supports varied generation and editing tasks and improves benchmark performance over prior models. Its main limitation is imprecise control of object counts described numerically in text.

  • Problem

    Remote sensing image-text datasets are small and geographically or scenically limited, while existing methods struggle with global-scale, multi-resolution, and unbounded generation.

  • Method

    The paper builds Git-10M, a 10.5-million-pair global dataset with resolution metadata, and trains Text2Earth, a 1.3-billion-parameter diffusion foundation model.

  • Results

    Text2Earth performs zero-shot text2image generation and generalizes across unbounded scene construction, image editing, and cross-modal generation, with +26.23 FID and +20.95% Zero-shot Cls-OA over prior models.

  • Takeaways & Limitations

    Git-10M and Text2Earth support global-scale, resolution-controllable remote sensing image generation beyond fixed-size and limited-scene models.

  • Takeaways & Limitations

    Text2Earth struggles to precisely control object counts specified in text, especially for larger quantities.

Abstract

from arXiv · show

Generative foundation models have advanced large-scale text-driven natural image generation, becoming a prominent research trend across various vertical domains. However, in the remote sensing field, there is still a lack of research on large-scale text-to-image (text2image) generation technology. Existing remote sensing image-text datasets are small in scale and confined to specific geographic areas and scene types. Besides, existing text2image methods have struggled to achieve global-scale, multi-resolution controllable, and unbounded image generation. To address these challenges, this paper presents two key contributions: the Git-10M dataset and the Text2Earth foundation model. Git-10M is a global-scale image-text dataset comprising 10.5 million image-text pairs, 5 times larger than the previous largest one. The dataset covers a wide range of geographic scenes and contains resolution information, significantly surpassing existing datasets in both size and diversity. Building on Git-10M, we propose Text2Earth, a 1.3 billion parameter generative foundation model based on the diffusion framework to model global-scale remote sensing scenes. Text2Earth integrates a resolution guidance mechanism, enabling users to specify image resolutions. A dynamic condition adaptation strategy is proposed for training and inference to improve image quality. Text2Earth excels in zero-shot text2image generation and demonstrates robust generalization and flexibility across multiple tasks, including unbounded scene construction, image editing, and cross-modal image generation. This robust capability surpasses previous models restricted to the basic fixed size and limited scene types. On the previous benchmark dataset, Text2Earth outperforms previous models with an improvement of +26.23 FID and +20.95% Zero-shot Cls-OA metric.Our project page is https://chen-yang-liu.github.io/Text2Earth

I. Introduction

Remote sensing text2image research lacks large, diverse datasets and models supporting global-scale, multi-resolution, unbounded generation. The paper addresses these gaps with Git-10M and the Text2Earth diffusion foundation model.

  • Research gaps: Existing remote sensing image-text datasets are small, geographically narrow, scene-limited, and often lack resolution metadata.
  • Dataset contribution: Git-10M contains 10.5 million global image-text pairs, five times larger than the previous largest dataset, with geographic and resolution metadata.
  • Model contribution: Text2Earth is a diffusion-based generative foundation model designed for global-scale remote sensing scenes and large-scale image synthesis.The model uses a VAE for compact feature-space diffusion and an OpenCLIP ViT-H encoder for text representation.
  • Capabilities: Text2Earth generalizes across zero-shot generation, unbounded scene construction, image editing, and cross-modal image generation.
  • Capabilities: Text2Earth supports user-guided multi-resolution zero-shot text2image generation across diverse global geographic scenes.
  • Evaluation: +26.23 FID and +20.95% Zero-shot Cls-OA improvements are reported on the RSICD benchmark over previous models.

A. Generative Foundation Models in the Computer Vision

Generative foundation models have progressed through GAN, autoregressive, and diffusion-based approaches for text-driven image generation. Remote sensing applications remain constrained by limited datasets and difficulty modeling complex global geographic structure.

  • GAN-based models: GAN-based text2image models use adversarial generator-discriminator training, with later methods adding staged refinement, attention, or structural information.
  • Autoregressive models: Autoregressive models generate image pixels or regions sequentially with Transformer architectures conditioned on text and preceding outputs.
  • Diffusion-based models: Diffusion models corrupt data with noise and learn a reverse denoising process, offering stable training and diverse photorealistic generation.
  • Diffusion-based models: Recent diffusion systems add conditioning and interaction mechanisms for spatial control, editing, and other controllable generation tasks.
  • Remote sensing applications: Remote sensing text2image methods evolved from GANs and autoregressive models toward diffusion, but generated quality and controllability remain challenging.
  • Open challenges: Limited diverse training datasets hinder modeling complex global geographic features and generalization across unbounded scene construction and image editing.

III. Global-scale image-text Dataset

Git-10M is a global-scale remote sensing image-text dataset built through diverse data collection, geographic expansion, filtering, enhancement, and automated annotation. The resulting resource combines large scale with broad geographic coverage, varied resolutions, and enriched textual descriptions.

  • Git-10M contains 10.5 million remote sensing image-text pairs with geographic locations and resolution information.
  • The dataset combines public remote sensing datasets with globally collected Google Earth imagery and preserves scene labels for text annotation.
  • Global sampling and targeted selection cover typical geographic features including urban areas, forests, mountains, and deserts while retaining location and resolution metadata.
  • Duplicate ocean scenes were removed, and image enhancement addressed noise and artifacts that could harm generative-model training.
  • Git-10M achieves improvements in scale, quality, diversity, and geographical coverage through its multi-stage collection and processing workflow.
  • The automated annotation pipeline uses GPT-4o with prompt optimization, metadata context, auditing, and manual sampling to produce semantically rich descriptions.

C. Dataset Analysis

Dataset analysis shows that Git-10M spans diverse geographic scenes and resolutions, with enhancement improving image quality and text descriptions averaging approximately 52 words. The analysis also introduces Text2Earth’s diffusion-based architecture for text-consistent generation and masked local editing.

  • Geographical Coverage: Git-10M spans multiple continents and regions, covering urban areas, forests, mountains, deserts, and other typical scenes.
  • Image Evaluation: Image quality improves significantly after enhancement processing, providing more reliable training data for generative models.
  • Resolution Distribution: The dataset ranges from 0.5m/pixel to 128m/pixel, supporting both detailed fine-grained imagery and broader-area coverage.
  • Text Analysis: Git-10M contains over 10.5 million text samples and more than 5.5 billion words, with descriptions averaging approximately 52 words.
  • Text2Earth Foundation Model: Text2Earth is a 1.3 billion parameter diffusion foundation model built from image compression encoding, conditional embedding, and diffusion modelling.
  • Text2Earth Foundation Model: Text2Earth generates new images consistent with text and resolution or performs masked local editing while preserving existing image structure.
  • Image Compression Encoding: The VAE compresses high-resolution pixels into a compact implicit space, improving computational efficiency for subsequent diffusion modelling.

2) Diffusion Modeling:

Text2Earth performs diffusion modeling in a VAE-compressed feature space, conditioning denoising on text, resolution, and masked-image information for controllable generation and editing.

  • 2) Diffusion Modeling:: The reverse diffusion process denoises the implicit representation and reconstructs the original image by predicting the Gaussian noise component.The U-Net denoising network is trained to predict noise during reconstruction.
  • 2) Diffusion Modeling:: Text2Earth performs diffusion modeling in a VAE-compressed feature space to reduce computational requirements while preserving image fidelity.This design supports large-scale and unbounded remote sensing image generation.
  • 2) Diffusion Modeling:: Text2Earth encodes text with OpenCLIP ViT-H and injects the resulting semantic embedding into U-Net layers through cross-attention.This mechanism dynamically focuses on critical semantic features to preserve textual faithfulness.
  • 2) Diffusion Modeling:: Resolution information is projected into an embedding, combined with the timestep embedding, and supplied to the U-Net to adjust image resolution during diffusion.The mechanism enables flexible, precise resolution control.
  • 2) Diffusion Modeling:: A masked-image encoding mechanism concatenates encoded masked-image features with diffusion features, enabling local editing while preserving original structure.The joint conditional representation is passed to the denoising U-Net for noise prediction.

B. Dynamic Condition Adaptation Strategy

Dynamic Condition Adaptation improves robustness by randomly dropping text and resolution conditions during training, preparing Text2Earth to operate with incomplete inputs.

  • B. Dynamic Condition Adaptation Strategy: Dynamic Condition Adaptation randomly drops text and resolution conditions during training to improve robustness to missing or unreliable inputs.The strategy combines conditional and unconditional learning so the model learns denoising dynamics under different condition availability.

2) Sampling with Scalable Condition Guidance:

During sampling, Text2Earth combines conditional and null-condition predictions to control denoising, balancing alignment with the requested inputs against image diversity and quality.

  • 2) Sampling with Scalable Condition Guidance:: During sampling, the DCA strategy mixes conditional and null-condition inputs to refine denoising toward the desired conditions.The approach predicts conditioned and unconditioned noise at each denoising step.
  • 2) Sampling with Scalable Condition Guidance:: The guidance scale ω controls how strongly the sampling process relies on the provided conditions.The final predicted noise is a weighted combination of conditional and unconditioned predictions.
  • 2) Sampling with Scalable Condition Guidance:: Scalable condition guidance enables Text2Earth to handle incomplete inputs while maintaining alignment, diversity, and image quality.The strategy is intended for varied input scenarios such as missing text or resolution conditions.

B. Implementation Details

Implementation uses large-scale distributed training, progressive refinement, specialized model variants, and several metrics and benchmark comparisons to evaluate Text2Earth.

  • B. Implementation Details: Training used 8 NVIDIA A100 GPUs, AdamW with a 0.0001 learning rate, batch size 1024, and 256 × 256 generated images.
  • B. Implementation Details: Progressive training first used the complete Git-10M dataset and then fine-tuned on samples scoring above 4.8 to improve fidelity and detail.
  • B. Implementation Details: Text2Eartht targets text-and-resolution generation, while Text2Earthe is specialized for image editing.The two variants address distinct remote sensing tasks.
  • C. Evaluation Metrics: Evaluation includes FID, Zero-Shot classification Overall Accuracy, and CLIP score to assess generated-image quality, semantic alignment, and text-image similarity.FID compares real and generated feature distributions, while CLIP score averages cosine similarity across text-image pairs.
  • B. Implementation Details: Table II compares Text2Earth with previous text2image methods on the RSICD dataset.

D. Zero-Shot Text2Image Generation

Text2Earth generates diverse remote sensing scenes from text and remains robust when text or resolution conditions are incomplete. It also produces resolution-dependent spatial detail and transfers effectively to RSICD through LoRA fine-tuning.

  • Text2Earth generates diverse geographical scenes, including mountains, rivers, urban areas, forests, and farmland, from user-free text without scene-specific fine-tuning.
  • Text2Earth can generate realistic scenes when text or resolution conditions are missing, demonstrating robustness to incomplete inputs.This behavior benefits from dynamic condition adaptation.
  • 0.5m, 1m, and 2m per pixel resolutions produce corresponding changes in storage-tank scale, density, and scene detail under the same prompt.
  • Resolution-dependent tree structure detail further demonstrates controllable generation across spatial scales.
  • +26.23 FID and +20.95% Zero-Shot Classification OA are achieved on RSICD after LoRA fine-tuning, outperforming previous methods.At guidance scale factor ω=3.0, the model achieves a favorable FID and Zero-shot Cls-OA trade-off.

E. Remote Sensing Image Editing

Text2Earth extends beyond text-to-image generation to editing, unbounded outpainting, and multimodal generation. Resolution control supports spatial coherence, while LoRA enables text-driven generation across remote sensing modalities.

  • Remote Sensing Image Editing: Text2Earth edits remote sensing content using text and masks, including cloud removal, feature replacement, and adding geographic elements.
  • Remote Sensing Image Editing: The edited regions integrate seamlessly with surrounding areas, maintaining continuity and coherence for customized applications.
  • Unbounded Scene Construction: Iterative outpainting enables seamless and effectively infinite expansion of remote sensing scenes on a canvas, overcoming fixed-size limitations.
  • Unbounded Scene Construction: Using the same resolution at each outpainting step preserves consistent spatial detail and visual coherence across expanded regions.
  • Text-Driven Multi-Modal Image Generation: LoRA fine-tuning enables Text2Earth to generate RGB, SAR, NIR, and PAN images with high quality and semantic consistency.Generated NIR images assign high pixel values to green vegetation, aligning with NIR imaging principles.
  • Text-Driven Multi-Modal Image Generation: FID scores across modalities are not directly comparable because each modality uses an Inception V3 model pretrained on modality-specific data.
  • Text-Driven Multi-Modal Image Generation: SAR scene classification is inherently more challenging than classification of RGB, NIR, or PAN images because SAR lacks comparable color and texture information.

2) Image-to-Image Translation:

Text2Earth supports cross-modal translation and image enhancement by conditioning a frozen generative model with a trainable modality encoder. Experiments show high-fidelity translation and downstream benefits from synthetic-image augmentation and Git-10M-based vision-language pretraining.

  • Image-to-Image Translation: Text2Earth covers PAN2RGB, NIR2RGB, PAN2NIR, super-resolution, and image dehazing for image-to-image translation and enhancement.
  • Image-to-Image Translation: A trainable ControlNet-inspired module encodes the conditional input modality while the frozen Text2Earth generates the target modality.
  • Image-to-Image Translation: Experiments on RSICD demonstrate high-fidelity translation across PAN2RGB, NIR2RGB, and PAN2SAR, while preserving physical correspondences such as vegetation regions.
  • Data Augmentation: Synthetic images generated from 1,027 RSICD text-image-category triplets are used to augment training for four remote sensing classification models.The experiment generated over 20,000 synthetic images and compared training with and without augmentation.
  • Vision-Language Pretraining: Git-RSCLIP is pretrained on Git-10M with contrastive learning and evaluated through zero-shot classification on multiple remote sensing datasets.

VI. Limitation and Discussion for Text2Earth Model

Text2Earth demonstrates broad generative capabilities, but precise object-count control remains limited. The paper identifies broader applications, autoregressive models, and more diverse multimodal datasets as future directions.

  • Limitations: Text2Earth struggles with fine-grained object-count control, generating nine rather than twelve storage tanks and eight rather than seven farmlands in examples.
  • Limitations: The authors attribute quantity-control difficulty to aligning textual numerical information with spatial visual content and emphasize high-level semantic learning.
  • Conclusion: Text2Earth can perform zero-shot image generation, image editing, unbounded scene construction, text-driven multimodal generation, and cross-modal generation.
  • Future Work: Future work could adapt Text2Earth to image enhancement, object detection, and change detection.
  • Future Work: Future research could train higher-capacity autoregressive remote sensing foundation models using Git-10M.
  • Future Work: Future datasets could pair text with SAR, NIR, hyperspectral, and other modalities to advance cross-modal generation and multimodal learning.
  • Conclusion: +26.23 FID and +20.95% Zero-shot Cls-OA are reported on the previous benchmark dataset relative to previous models.
Loading 2501.00895v2…