PaperlayerView exact revision on arXiv

Source-linked AI summary

MEOX: Compact Multimodal Mixture-of-Experts for Earth Observation

Mohanad Albughdadi

arXiv:2609.05351v1cs.CV

TL;DR

Earth Observation representation learning must handle heterogeneous sensors, missing observations, variable spatial grids, and practical model-size constraints. MEOX addresses this with a compact multimodal masked autoencoder using validity-aware sensor processing, delayed fusion, sparse experts, and metadata tokens. It achieves strong frozen and adapted transfer alongside useful same-sensor retrieval, while cross-sensor alignment and several robustness evaluations remain open.

  • Problem

    Earth Observation encoders must accommodate complementary heterogeneous sensors, missing inputs, differing spatial grids, and model-size constraints.

  • Method

    MEOX combines independent validity-aware sensor processing, learned delayed fusion, metadata tokens, and partially shared sparse experts in a compact multimodal masked autoencoder.

  • Results

    MEOX provides useful frozen transfer across GEO-Bench tasks, reaches 72.95% BigEarthNet micro-mAP after adaptation, and separates same-sensor semantics from cross-sensor alignment in retrieval.

  • Takeaways & Limitations

    The results support sensor-flexible representation learning and inspectable sparse computation within a compact parameter budget.

  • Takeaways & Limitations

    The current evidence does not establish random-band-removal robustness, matched component gains, or large-scene overlapping-window inference.

Abstract

from arXiv · show

Recent advances in Earth Observation representation learning accommodate heterogeneous sensors and missing observations, often through larger architectures. We present MEOX (Multimodal Earth Observation with eXperts), a multimodal masked autoencoder with a 2.939 million-parameter encoder and 3.115 million parameters in total. Sensor-specific adapters, explicit validity signals, and a shared sparse-expert block preserve modality-dependent processing before a learned patch-wise fusion. Four metadata tokens then accompany a single spatial sequence through fourteen further encoder blocks. Shared expert projections with private low-rank residuals constrain parameter growth, while rotary attention supports downstream spatial grids different from pretraining. The model is pretrained on 1.228 million MMEarth64 samples using modality-balanced masked reconstruction and structured sensor dropout. Frozen transfer is evaluated on six GEO-Bench tasks at both 64 and 224 pixels. The model reaches 64.42% mean intersection-over-union on cashew segmentation at 64 pixels and 90.56% average accuracy on EuroSAT at 224 pixels, exceeding the corresponding reported CSMoE results. BigEarthNet finetuning reaches 72.95% micro-average precision. Routing diagnostics distinguish expert participation, spatial dependence, modality association, and functional contribution. A held-out WorldCover probe measures a 0.64-percentage-point benefit from metadata, while retrieval separates same-sensor semantics from cross-sensor alignment. These results demonstrate sensor-flexible representation learning and strong task transfer using a compact parameter budget.

I. INTRODUCTION

MEOX addresses the challenge of retaining multimodal flexibility and inspectable specialization in a compact Earth Observation encoder. It combines validity-aware sensor processing, delayed fusion, sparse experts, and broad evaluation to study the resulting accuracy–resource trade-off.

  • Motivation: EO inputs combine complementary optical, SAR, and metadata information, but archives commonly contain heterogeneous sensors, missing observations, and varying spatial grids.The model must distinguish missing inputs from valid numerical observations while remaining practical for large archives or limited hardware.
  • Sparse experts: Sparse routing requires downstream performance and direct route analysis because active experts may remain functionally similar or exhibit imbalanced discrete dispatch.MEOX therefore evaluates actual expert participation and specialization alongside task transfer.
  • Architecture: MEOX extends Geo-MoE-MAE into a Sentinel-1/Sentinel-2 representation learner with shared sensor processing, validity-aware delayed fusion, metadata tokens, and partially shared sparse experts.Learned patch-wise fusion reduces independently processed sensor streams to one sequence before deeper encoding.
  • Contributions: 2.939M parameters define the encoder, combining independent shared-weight sensor processing, explicit missingness, balanced reconstruction of omitted sensors, and deterministic variable-grid inference.The design uses a compact multimodal architecture rather than carrying separate modality sequences through the entire encoder.
  • Related foundations: MEOX builds on masked autoencoders, multimodal reconstruction, sparse mixture-of-experts, rotary attention, and compact staged expert backbones from prior work.The paper positions its contribution as a joint multimodal architecture and measured accuracy–resource trade-off.

A. Inputs and Missingness

MEOX preserves sensor-specific information and explicit missingness before combining available inputs into one spatial sequence. Metadata missing states and validity-aware fusion make absent or partially valid observations distinguishable from genuine values.

  • Inputs and missingness: Missing raster elements are zero-filled only after validity is retained, so a missing value differs from a valid observation equal to its normalization mean.Band-presence vectors also identify which configured channels are supplied.
  • Sensor tokens: Each sensor patch combines content, validity-mask, sensor-identity, and band-presence embeddings into one sensor-aware token.The validity projection is initialized to zero, preserving the initial image representation while enabling missingness modeling during training.
  • Metadata tokens: Metadata uses four tokens for latitude, longitude, month, and ERA5 climate context, with explicit missing-state contributions when components are unavailable.Omitting metadata is represented as missing data rather than as a genuine zero value.
  • Shared sensor processing: Before fusion, the same transformer-MoE block processes each available sensor independently, allowing attention and routing to operate on sensor-specific observations.This preserves modality-dependent processing without carrying separate sensor sequences through the deeper encoder.
  • Delayed fusion: Fusion weights combine available sensor representations at each spatial position using learned scores and a logarithmic validity prior.The softmax excludes unsupplied sensors, weights sum to one, and lower valid fractions receive increasingly negative validity contributions.
  • Post-fusion encoding: After fusion, four metadata tokens and CLS accompany one spatial sequence through fourteen additional transformer-MoE blocks.The deeper encoder processes one fused stream rather than a separate sequence for each sensor.

C. Compact Sparse Experts

MEOX uses compact sparse experts by sharing major projections within each layer while retaining private low-rank residuals for expert-specific adaptation. Noisy top-2 routing selects experts during training and deterministic routing is used at evaluation.

  • Expert layout: MEOX uses three, four, and five experts across early, middle, and late encoder layers while keeping token width fixed at d = 144.Expert hidden width decreases from 144 in the first layer to 72 in the last, reducing parameter growth with depth.
  • Routing: Training adds input-dependent Gaussian routing noise, while evaluation uses deterministic routing logits.The noise scale is positive and bounded away from zero by δ = 10^-9.
  • Sparse dispatch: Top-2 routing selects the two experts with the largest logits, assigns them positive weights summing to one, and gives unselected experts zero weight.Both selected experts are evaluated, and MEOX imposes no per-expert capacity limit, so tokens are not dropped under heavy assignment.
  • Shared projections: Each expert is a SwiGLU network whose value and output projections are shared within a layer rather than fully duplicated across experts.Expert-specific behavior is introduced through private residual paths instead of complete private copies.
  • Low-rank residuals: Rank-ρ private residual paths modify shared value and output transformations without storing complete private projections.The U matrices start at zero, so expert-specific residual contributions emerge during training; here ρ = 8.

D. Position and Reconstruction Decoder

Axial rotary attention supplies runtime spatial coordinates without adding absolute-position vectors to the residual stream. During masked pretraining, a compact decoder restores masked positions and reconstructs sensor targets from the encoder representation.

  • Position encoding: Axial two-dimensional RoPE rotates attention queries and keys using runtime patch coordinates, supporting rectangular inputs and variable token grids.Both image dimensions must be divisible by the patch size, and RoPE does not by itself establish invariance to physical ground-sampling distance.
  • Masked reconstruction: 75% of spatial patches are masked during pretraining using identical mask indices across sensors, leaving 64 visible patches at 64 pixels.The encoder output is projected to width 108, visible positions are restored, and learned mask tokens fill the masked locations.
  • Reconstruction decoder: Two dense decoder blocks with RoPE and SwiGLU predict all patch elements through separate sensor heads.The decoder has 176,096 parameters, or 5.65% of the total model parameters.

E. Pretraining Objective

MEOX pretraining balances reconstruction across sensors, trains recovery from structured sensor dropout, and regularizes sparse-expert routing. Validity-aware masking ensures that only valid masked targets contribute to reconstruction.

  • Structured sensor dropout: Sensor dropout omits input combinations while retaining all three sensor streams as reconstruction targets.The six combinations include single-sensor and two-sensor inputs; each sensor is absent in three combinations.
  • Validity-aware masking: Reconstruction averages squared error over valid elements in masked patches, excluding visible patches and invalid targets.A maximum denominator of one prevents division by zero when a sensor has no valid masked target.
  • Modality-balanced reconstruction: Each sensor contributes one averaged loss, preventing Sentinel-2’s 13 bands from outweighing either two-band Sentinel-1 target.Without normalization, Sentinel-2 would contribute approximately 13/2 = 6.5 times as many error terms.
  • Expert routing: A balance loss regularizes top-2 expert assignments, increasing when router probabilities and executed assignments concentrate on few experts.Executed assignment fractions are detached from gradients because top-2 selection is discrete.
  • Expert routing: Balance losses are averaged across sensor-specific layer-0 evaluations and all 15 encoder layers.The objective excludes position penalties, dispatch guards, latent alignment, visible-patch reconstruction, and auxiliary semantic losses.

A. MMEarth64 Pretraining

Pretraining uses a large multimodal Earth Observation corpus with specified optimization and validation procedures. Frozen transfer evaluates configured inputs under native-grid and standard-grid spatial protocols.

  • Dataset and optimization: 1,228,121 training and 12,405 validation samples come from a deterministic 1% holdout of MMEarth64.The corpus uses Sentinel-2, ascending and descending Sentinel-1 VV/VH, and four metadata groups.
  • Dataset and optimization: AdamW training runs for 50 epochs with batch size 128, peak learning rate 3 × 10^-4, weight decay 0.05, and cosine decay after warmup.Gradient clipping, mixed precision, deterministic routing, and fixed-seed repeated-mask validation are also used.
  • Frozen transfer: Frozen GEO-Bench transfer uses official splits and normalization statistics without training augmentation in downstream heads.All physically available bands compatible with the checkpoint are used.
  • Spatial protocols: Native-grid 64 resizes imagery to pretraining dimensions, while standard-grid 224 uses 224 × 224 imagery.Segmentation retains 224 × 224 ground-truth class maps and evaluates logits at that resolution.

C. Comparison and Analysis Controls

MEOX is evaluated with controlled frozen-transfer comparisons, scale accounting, and routing diagnostics. The analyses show strong task-dependent transfer and substantial parameter savings, while expert routing is active, sensor-associated, semantically related, and functionally consequential.

  • Comparison controls: CSMoE provides the closest published comparison, but the evaluation matches tasks and metrics rather than every band, pretraining distribution, or implementation choice.Competing models were not rerun, and other-model values digitized from comparison figures are approximate.
  • Frozen transfer: 64-pixel transfer reaches 89.70% AA on EuroSAT and 64.42% mIoU on cashew, exceeding reported CSMoE variants by 1.40 and 5.02 points.The 224 grid improves four tasks, most notably crop type by 4.68 points, but changes cashew little and reduces So2Sat by 2.43 points.
  • Scale and computation: MEOX has approximately 87 times fewer parameters than CSMoE P = 14, while native-grid S2 inference uses 2.84 times fewer counted operations than CSMoE P = 32.At 224 pixels, global attention raises counted operations to 51.477G, so the compute advantage does not persist despite the parameter advantage.
  • Routing diagnostics: Across the 128-sample routing diagnostic, every expert receives tokens, while global routing is balanced but individual-token assignments remain selective.Minimum execution rate, gate share, and top-1 rate are 25.12%, 11.55%, and 9.28%; normalized entropy is 0.984–0.999 globally versus 0.886–0.956 per token.
  • Routing diagnostics: Routing shows little fixed spatial dependence but clear sensor and land-cover association, with expert outputs becoming more differentiated in deeper layers.Position-to-expert NMI ranges from 0.00189 to 0.00623 near shuffled baselines, whereas Dynamic World class/top-1-expert NMI ranges from 0.0270 to 0.1232; output cosine similarities are approximately 0.68–0.89 early and 0.39–0.69 deeper.
  • Functional contribution: Suppressing executed experts raises reconstruction loss, with larger effects in layer 0 and for expert 0 at layer 14.For a fixed masked batch, layer-0 suppression increases loss by 0.00215–0.00334, while layer-14 expert 0 increases it by 0.00325.

D. Embedding Selection and Geometry

The EuroSAT pilot selects pre-norm mean-fine features for dense prediction, while geometry analysis finds concentrated variance whose dominant directions remain task-relevant.

  • Representation selection: 89.68% average accuracy is achieved by averaging all pre-norm tokens, while pre-norm mean-fine features are selected within the predefined 0.5-point tie threshold.The selected representation excludes CLS and metadata tokens and directly corresponds to spatial features used for dense prediction.
  • Embedding geometry: 8.44 out of 144 is the effective rank for pre-norm features, compared with 6.18 for post-norm features on 1,000 EuroSAT validation images.Raw mean pairwise cosine similarity is 0.5406 for pre-norm and 0.5347 for post-norm features, falling near zero after mean subtraction.
  • Embedding geometry: 15.00 and 15.61 are the effective ranks after per-dimension standardization, but more uniform geometry does not necessarily preserve task-relevant information.The paper therefore evaluates downstream probes rather than treating geometric uniformity as sufficient.
  • Embedding geometry: 89.0% probe accuracy at k = 0 falls to 70.1% at k = 8 after removing leading principal components, showing that dominant directions contain useful semantic information.Standardization modestly improves nearest-neighbor retrieval but does not outperform raw pre-norm features in the linear probe, so raw features are retained.
  • Input normalization: 90.60% validation average accuracy is obtained with MMEarth normalization, whereas official GEO-Bench statistics yield 89.52% and no z-scoring yields 61.36%.Main experiments nevertheless use official task statistics to preserve comparability with published results.

E. Metadata Availability on Held-Out WorldCover

The held-out WorldCover probe tests metadata availability using frozen image features and a small segmentation head. Metadata provides a modest positive benefit, with effects varying across metadata groups and classes.

  • 30.45% test mIoU is achieved with all metadata in the held-out WorldCover probe.The probe uses a 1,595-parameter head over frozen pre-norm features for 11 WorldCover classes at 64×64.
  • 0.64 points are lost when all metadata is removed using a matched head, while ERA5 has the largest isolated measured effect.Removing all metadata with the original head costs 0.72 points; these effects are not additive because metadata groups and image content can be redundant.
  • Nine of eleven classes improve with all metadata relative to none, and matched predictions disagree on 3.46% of valid pixels.Month removal has little effect, whereas ERA5 contributes the largest isolated measured change.

F. Task Adaptation and Retrieval

Task adaptation substantially improves BigEarthNet performance, while retrieval shows useful same-sensor semantics but incomplete radar–optical alignment. The broader evidence supports compact, inspectable transfer while leaving several multiscale and alignment questions open.

  • Task adaptation: 72.95% micro-mAP is reached on BigEarthNet at 224 pixels after task adaptation.Finetuning enables all encoder blocks while freezing 17,400 router parameters and uses spatial augmentation, distinct from frozen probing.
  • Task adaptation: Micro-mAP rises from 55.16% to 67.78% at 64 pixels and from 57.22% to 72.95% at 224 pixels.The adapted protocol retains the dataset split, band mapping, and pooling while adding trainable encoder capacity.
  • Retrieval: Same-sensor retrieval F1 reaches 64.41% for S1 and 66.33% for S2, while cross-sensor F1 is lower.The retrieval setup uses five-neighbor exact cosine search over paired optical/SAR images.
  • Retrieval: MEOX retrieval exceeds CSMAE and two coarser CSMoE variants for S1, but S2 and the strongest cross-sensor directions remain below the best references.The pattern indicates useful sensor-specific semantics without a fully aligned radar–optical metric space.
  • Overall implications: The central result is useful frozen transfer, flexible configured inputs, and inspectable sparse computation within a three-million-parameter system.The contribution combines shared sensor-specific processing, validity-aware delayed fusion, private low-rank expert residuals, metadata missing states, and balanced missing-input reconstruction.
  • Overall implications: 64-pixel execution is the efficient operating point, whereas 224-pixel execution demonstrates grid flexibility at higher attention cost.The empirical profile leaves crop segmentation, cross-sensor retrieval, random-band robustness, matched component ablations, and large-scene inference for future work.
Loading 2609.05351v1…