PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexiMo: A Flexible Remote Sensing Foundation Model

Xuyang Li, Chenyu Li, Pedram Ghamisi, Danfeng Hong

arXiv:2503.23844v1cs.CV

TL;DR

Fixed patch sizes and spatial configurations constrain remote-sensing models facing heterogeneous resolutions, modalities, and channel configurations. FlexiMo adapts patch embeddings spatially and uses wavelength-aware channel adaptation; the authors report that it outperforms existing methods across benchmarks and evaluates it on diverse image- and pixel-level tasks.

  • Problem

    Fixed patch tokenization and input configurations limit models' adaptation to varied resolutions, patch sizes, modalities, and sensor channels.

    Source: §I · pp. 1–2

  • Method

    FlexiMo combines spatially adaptive patch embeddings with a wavelength-aware design for variable spectral configurations.

    Source: §I · p. 2

  • Results

    FlexiMo consistently outperforms existing methods across all benchmarks and is validated on diverse datasets for image- and pixel-level tasks.

    Source: §I · p. 2, §V · p. 12

  • Takeaways & Limitations

    FlexiMo's resolution alignment preserves token characteristics and multi-scale feature fidelity without modifying the pretrained network architecture.

    Source: §V · p. 12

Abstract

from arXiv · show

The rapid expansion of multi-source satellite imagery drives innovation in Earth observation, opening unprecedented opportunities for Remote Sensing Foundation Models to harness diverse data. However, many existing models remain constrained by fixed spatial resolutions and patch sizes, limiting their ability to fully exploit the heterogeneous spatial characteristics inherent in satellite imagery. To address these challenges, we propose FlexiMo, a flexible remote sensing foundation model that endows the pre-trained model with the flexibility to adapt to arbitrary spatial resolutions. Central to FlexiMo is a spatial resolution-aware module that employs a parameter-free alignment embedding mechanism to dynamically recalibrate patch embeddings based on the input image's resolution and dimensions. This design not only preserves critical token characteristics and ensures multi-scale feature fidelity but also enables efficient feature extraction without requiring modifications to the underlying network architecture. In addition, FlexiMo incorporates a lightweight channel adaptation module that leverages prior spectral information from sensors. This mechanism allows the model to process images with varying numbers of channels while maintaining the data's intrinsic physical properties. Extensive experiments on diverse multimodal, multi-resolution, and multi-scale datasets demonstrate that FlexiMo significantly enhances model generalization and robustness. In particular, our method achieves outstanding performance across a range of downstream tasks, including scene classification, land cover classification, urban building segmentation, and cloud detection. By enabling parameter-efficient and physically consistent adaptation, FlexiMo paves the way for more adaptable and effective foundation models in real-world remote sensing applications.

I. INTRODUCTION

Remote sensing models often rely on fixed tokenization and input configurations, limiting adaptation to diverse resolutions, patch sizes, and sensor channels. FlexiMo combines spatially adaptive patch embeddings with wavelength-guided channel adaptation and is validated across diverse remote-sensing tasks.

I. INTRODUCTION

  • Fixed patch tokenization can disrupt positional geometry as resolution changes and forces a trade-off between high-resolution computation and low-resolution detail.Models tailored to particular modalities or fixed spectral bands can also struggle with differing downstream inputs and distribution shifts.Source: §I · pp. 1–2
  • Existing approaches commonly assume fixed spatial configurations or address channel variation without also handling flexible spatial resolution and patch size.Preprocessing can disturb physical data properties, while extensive fine-tuning requires substantial computation and risks overfitting in low-data settings.Source: §I · p. 2
  • FlexiMo adapts patch embeddings to input resolution and patch size through a spatial resolution-aware module, preserving token properties without changing the underlying architecture.The module uses pseudo-inverse bilinear interpolation; wavelength-guided channel adaptation generates kernels for variable spectral configurations while preserving spectral fidelity.Source: §I · p. 2
  • FlexiMo is presented as supporting arbitrary input resolution, size, and channel count while preserving spectral fidelity through wavelength-aware adaptation.The authors report validation across modalities, resolutions, and channel configurations, including RGB, multispectral, and SAR data, on image- and pixel-level tasks.Source: §I · p. 2

II. RELATED WORK

The section outlines ViT patch tokenization, positional embeddings, and self-attention, then reviews efforts to adapt vision architectures and remote-sensing models to varying image sizes. It identifies arbitrary combinations of image sizes and patch embeddings as an unresolved robustness challenge in the approaches discussed.

A. Development of Vision Transformer

  • ViT divides an image into non-overlapping patches, projects each flattened patch into an embedding, and adds positional embeddings before transformer encoding.The encoder uses a [CLS] token, self-attention layers, and feed-forward networks; attention uses query, key, and value matrices.Source: A. Development of Vision Transformer · p. 2
  • Architecture adaptations include hierarchical window attention, linear attention, and lightweight design, while FlexViT, ViTAR, and MSPE address varying image sizes or patch embeddings.The section notes that performance can decline when image size changes during fine-tuning or inference.Source: A. Development of Vision Transformer · p. 3
  • Remote-sensing spatial optimization approaches discussed are predominantly tested under fixed, known conditions and have not demonstrated robust performance across arbitrary image-size and patch-embedding combinations.This leaves a gap between handling selected spatial settings and generalizing across arbitrary combinations.Source: A. Development of Vision Transformer · p. 3

B. Remote Sensing Foundation Model

Remote-sensing foundation models span vision-only and vision-language approaches, but many use fixed input configurations. FlexiMo targets variable spatial scales by generating patch embeddings and aligning them to the input patch size before pretrained-model tokenization.

B. Remote Sensing Foundation Model

  • Vision-only remote-sensing models use self-supervised methods on single- or multi-sensor data, while vision-language models align imagery with text or enable interaction with language models.The section describes these model families as commonly pretrained with fixed modalities, image sizes, and channels, which can hinder processing or performance on differing downstream inputs.Source: B. Remote Sensing Foundation Model · pp. 3–4

A. Overview of the Proposed FlexiMo

  • FlexiMo is designed for unified perception across arbitrary spatial scales to adapt pretrained models to multi-scale remote-sensing imagery.Its inputs include an image, a patch-size parameter corresponding to resolution, and electromagnetic wavelength parameters; a dynamic generator produces patch embeddings that the spatial module aligns.Source: A. Overview of the Proposed FlexiMo · p. 4
  • The aligned embeddings are applied to the image before patchification into tokens for the pretrained model, with patch size adjusted to input characteristics.The method uses smaller patches for low-resolution or small images to capture finer details and larger patches for high-resolution or larger images to preserve global context and reduce redundancy.Source: A. Overview of the Proposed FlexiMo · p. 4

B. Dynamic Channel Adaptation with Wavelength Embedding

FlexiMo uses channel wavelengths as prior information to generate convolution kernels for variable-channel remote sensing inputs. This replaces fixed spectral configurations that can disrupt sensor data’s physical properties.

B. Dynamic Channel Adaptation with Wavelength Embedding

  • Fixed channel counts in conventional patch embedding can force grouping or padding, disrupting the physical spectral properties of remote sensing data.FlexiMo instead generates kernels from channel wavelengths to adapt to varying channel configurations.Source: B. Dynamic Channel Adaptation with… · p. 4
  • Each channel’s central wavelength is embedded, refined through a nonlinear transformation, and used as prior information for dynamic kernel generation.The wavelength vector represents the central wavelength of each channel; its embedding has dimension D.Source: B. Dynamic Channel Adaptation with… · pp. 4–5
  • A lightweight Transformer processes wavelength embeddings and generates convolution weights and biases tailored to the input’s spectral characteristics.The resulting weights and biases are reshaped into a convolution kernel and applied to the input for patch embedding.Source: B. Dynamic Channel Adaptation with… · p. 5
  • The dynamic weight generator requires extensive pre-training; without it, fine-tuning on small datasets may underperform a standard Transformer.The generator addresses channel flexibility, but its reliance on fixed image and patch sizes leaves spatial adaptability to subsequent methods.Source: B. Dynamic Channel Adaptation with… · p. 5

C. Spatial Resolution-Aware Module with Parameter-Free Alignment Embedding

FlexiMo’s parameter-free Spatial Resolution-Aware Module uses PI-Resize to align patch embeddings with remote-sensing spatial resolution while preserving embedding properties. It combines this spatial adaptation with channel adaptation in a foundation-model fine-tuning framework.

C. Spatial Resolution-Aware Module with Parameter-Free Alignment Embedding

  • For remote sensing, Ground Sample Distance measures the ground distance between adjacent pixel centers, making input perceptual geometry relevant to downstream tasks.The module dynamically adjusts patch size for arbitrary image resolutions to support tokenization and feature extraction.Source: C. Spatial Resolution-Aware Module… · pp. 5–6
  • The plug-and-play, parameter-free module uses a pseudo-inverse transformation to counter bilinear-interpolation distortion in patch embeddings.Directly interpolating embedding weights can dramatically change token norms; the pseudo-inverse restores the original scale when no information is lost.Source: C. Spatial Resolution-Aware Module… · p. 5
  • PI-Resize transforms the original patch-embedding kernel into a target-resolution kernel using a bilinear interpolation operator and its associated matrix.The original and new kernels are specified with spatial dimensions H×W and H′×W′, respectively.Source: C. Spatial Resolution-Aware Module… · p. 5
  • In upsampling, the stated choice precisely recovers the original token embedding; in downsampling, PI-Resize provides a principled mechanism to minimize information loss.For downsampling, the optimal solution depends on the underlying patch distribution.Source: C. Spatial Resolution-Aware Module… · p. 6
  • Unlike FlexiViT, FlexiMo targets remote-sensing resolution, integrates PI-Resize into foundation-model adaptation, and combines spatial and channel adaptation.The module adjusts patch size to the input resolution and captures multi-scale information.Source: C. Spatial Resolution-Aware Module… · p. 6

IV. EXPERIMENTS

The experiments evaluate FlexiMo after fine-tuning on image- and pixel-level remote-sensing tasks, alongside ablation studies.

IV. EXPERIMENTS

  • The evaluation covers medium-resolution MSI and SAR scene classification, MSI land-cover classification, and high-resolution RGB building detection and cloud detection.The experiments are described as validating the proposed method across these downstream tasks.Source: §IV · p. 6
  • The experiments include both image-level scene classification and pixel-level land-cover, building, and cloud tasks.The paper groups these evaluations by resolution and modality in its task overview.Source: §IV · p. 6
  • Ablation studies are conducted to further substantiate the efficacy of the approach.Source: §IV · p. 6

1) Experimental Setup and Dataset Information:

The evaluation uses five remote-sensing datasets and fine-tunes a DOFA-based ViT-Base for downstream tasks. It reports OA for scene classification and OA, precision, recall, and IoU for pixel-level tasks.

1) Experimental Setup and Dataset Information:

  • Experiments use EuroSAT and EuroSAT-SAR for image-level tasks, and SegMunich, WHU-Building, and GF12MS-WHU for pixel-level tasks.Image-level experiments run on two GPUs and pixel-level experiments on four; the stated hardware is RTX 4090 GPUs and AMD EPYC 7Y83 CPUs.Source: 1) Experimental Setup and Dataset… · p. 6

2) Model Fine-Tuning and Task-Specific Adaptation:

  • The model uses DOFA ViT-Base pre-trained at 224×224 images and 16×16 patches, then fully fine-tunes all parameters for each downstream task.Image-level tasks use a [CLS]-token classification head, while pixel-level tasks use a UperNet segmentation head.Source: 2) Model Fine-Tuning and Task-Specific… · p. 6

3) Model Evaluation Metrics:

  • Scene classification is evaluated with OA, while pixel-level tasks use OA, precision, recall, and IoU.The metric definitions identify TP, TN, FP, and FN as true/false positives and negatives.Source: 3) Model Evaluation Metrics · p. 6

1) Performance on EuroSAT Dataset:

FlexiMo’s performance is evaluated on multispectral EuroSAT, SAR EuroSAT-SAR, and land-cover SegMunich datasets. Results report strong performance across these settings, including gains from smaller patches on EuroSAT-SAR and SegMunich.

1) Performance on EuroSAT Dataset:

2) Performance on EuroSAT-SAR Dataset:

1) Performance on SegMunich Dataset:

  • On the SegMunich dataset, the proposed method achieves state-of-the-art performance, outperforming the second-best method by 1.4 in terms of mIoU, as reported in Table IV.Source: 1) Performance on SegMunich Dataset · p. 8

2) Performance on WHU-Building Dataset:

On high-resolution building extraction and cloud detection, FlexiMo achieves the reported best performance on both datasets. The experiments also report improved fine-detail delineation as patch size decreases.

2) Performance on WHU-Building Dataset:

3) Performance on GF12MS-WHU Dataset:

D. Ablation Studies

The ablations examine image size, patch size, and channel configurations across datasets. They report robust performance across image sizes and improved dense-prediction performance with finer patch embeddings, alongside computational trade-offs.

D. Ablation Studies

  • The ablations cover multiple source datasets and vary image size, patch size, and channel count to assess downstream performance.Table IX is identified as the channel-configuration ablation on EuroSAT and EuroSAT-SAR.Source: D. Ablation Studies · p. 9, Table IX · p. 11

1) Ablation study on fine-tuning with different image and patch sizes:

2) Ablation study on pixel-level fine-tuning tasks with varying patch sizes:

3) Ablation study on fine-tuning with different channel configurations:

FlexiMo outperforms existing methods across the reported benchmarks, while ablations show gains across channel configurations and resizing strategies. Its spatial resolution-aware design adjusts patch embeddings without modifying the underlying network architecture.

3) Ablation study on fine-tuning with different channel configurations:

4) Ablation study on fine-tuning with various resizing strategies:

V. CONCLUSION

  • FlexiMo’s spatial resolution-aware module recalibrates patch embeddings to preserve token characteristics and multi-scale feature fidelity without changing the underlying network architecture.Source: §V · p. 12
  • FlexiMo outperforms existing methods across all benchmarks; ablations confirm robustness across image sizes, patch sizes, and channel configurations.Source: §V · p. 12
Loading 2503.23844v1…