PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing

Paolo Fraccaro, Gabby Nyirjesy, Daniela Szwarcman, Himanshu Patil, Vishal Gaur, Rohit Lal, Rachel A. Slank, Geoffrey Dawson, Hiyam Debary, Michael K. Barker, Andrew Annex, Vishnu Viswanathan, Zachary Morse, Ethan I. Schaefer, Nikolaos Dionelis, Ankur Kumar, Campbell D. Watson, Manil Maskey, Rebekah I. Dawson-Rigas, Juan Bernabé-Moreno, Rahul Ramachandran, Sujit Roy

arXiv:2609.13283v1cs.CVcs.AIcs.LG

TL;DR

Lunar remote sensing must reconcile instrument-specific measurements, illumination effects, and spatial scales spanning orders of magnitude. NASA-IBM LFM adapts multimodal masked-token pretraining with explicit acquisition geometry and joint training across two lunar scales. Across four benchmarks it matches or exceeds ImageNet baselines and improves over random initialization, with especially strong polar ice prospectivity results.

  • Problem

    Lunar observations span heterogeneous modalities and resolutions, while pixel reconstruction can waste capacity on noisy detail and leave illumination confounded with surface appearance.

    Source: §1 · p. 2

  • Method

    NASA-IBM LFM adapts multimodal masked-token pretraining with explicit acquisition-geometry context and joint training of WAC- and NAC-scale data.

    Source: §3.1 · p. 4, §6 · p. 19

  • Results

    Across four benchmarks, LFM matches or exceeds ImageNet baselines and improves over random-init; ice prospectivity reaches RMSE 0.0293±0.0013 versus the best baseline’s 0.0377±0.0004.

    Source: §1 · p. 3, §6 · pp. 20–21

  • Takeaways & Limitations

    LoRA is competitive on crater detection and IMP segmentation with fewer trainable parameters, while full fine-tuning performs best for ice prospectivity.

    Source: §6 · p. 21

    The generation experiments show that the model does not maintain a geodetic reference frame, limiting applications requiring absolute positioning.

    Source: §5.1 · p. 14, §6 · p. 21

Abstract

from arXiv · show

We present a multimodal foundation model for lunar remote sensing, pretrained from scratch on SomBench, a geographically partitioned corpus of nearly two million co-registered tile bundles spanning 11 modalities at two spatial scales (1 m/pixel and 100 m/pixel). The model adapts the TerraMind masked-token architecture with two lunar-specific extensions: acquisition geometry is provided as explicit context, and meter- and hundred-meter-scale tiles are trained jointly so that a single set of weights covers both resolutions. FlexiViT patch embeddings allow adaptation to different patch sizes without retraining, while modality-wise inputs enable flexible multimodal fine-tuning. Qualitative generation experiments suggest the model learns meaningful cross-modal correspondences, including terrain derivatives from elevation and illumination-consistent reflectance from geometry. We evaluate on four benchmarks: crater detection at WAC and NAC scales, irregular mare patch (IMP) segmentation, and polar ice prospectivity regression. Across tasks, the pretrained model matches or outperforms ImageNet-pretrained baselines and an architecturally identical random-init control. On multimodal ice prospectivity regression, pretrained variants achieve the best results, while the random-init model outperforms most baselines, suggesting gains arise from both the architecture and pretraining. Label efficiency is notable for WAC crater detection, where the pretrained model trained on 50% of the data exceeds the strongest ImageNet baseline trained on the full dataset. Among adaptation strategies, LoRA matches or surpasses full fine-tuning on crater detection and IMP segmentation while using far fewer trainable parameters, whereas full fine-tuning performs best for ice prospectivity regression. We release the pretrained checkpoint, benchmark datasets, and fine-tuning code to support reproducible lunar AI research.

1 Introduction

Lunar remote sensing combines complementary instruments across wide resolution ranges, but lunar pretraining must handle modality alignment, missing observations, and cross-scale structure. The paper introduces geometry-conditioned, mixed-resolution NASA-IBM LFM pretraining and reports gains across four benchmarks.

1 Introduction

  • Lunar instruments encode complementary surface properties across approximately four orders of magnitude in native resolution, complicating reusable representation learning.The modalities capture morphology, composition, thermophysics, and subsurface structure; a model must align physical variables, tolerate absent observations, and preserve structure across scales.Source: §1 · p. 2
  • Pixel reconstruction can waste capacity on noisy high-frequency detail, while pixel-only learning must disentangle appearance from illumination and geometry.Training one resolution family at a time also misses how regional context and meter-scale detail constrain one another.Source: §1 · p. 2
  • NASA-IBM LFM is pretrained from scratch on SomBench, nearly two million geographically partitioned, co-registered tile bundles spanning regional and meter-scale analysis.The corpus provides WAC- and NAC-anchored tracks with multiple auxiliary products and geographic splits that prevent overlapping observations from crossing partitions.Source: §1 · p. 2
  • The model adapts TerraMind with acquisition geometry as encoder input and jointly trains WAC- and NAC-anchored tracks in one mixed-resolution loop.Modality-wise tokenization and FlexiViT patch-embedding resizing support differing input modalities and patch grids without backbone retraining.Source: §1 · pp. 2–3
  • Across four benchmarks, pretrained NASA-IBM LFM matches or exceeds ImageNet baselines and an architecture-matched random-init control.The benchmarks cover WAC and NAC crater detection, IMP segmentation, and polar ice prospectivity regression.Source: §1 · p. 3

2 Related Work

Prior work supplies masked reconstruction objectives, lunar foundation models, and task-specific lunar mapping methods. This paper positions its approach within those lines while using separate co-ingested terrain modalities for learned cross-modal relationships.

2 Related Work

  • Masked autoencoding reconstructs hidden patches, while MultiMAE extends reconstruction across input and output modalities using a limited visible-token budget.TerraMind uses VQ-VAE discrete codes as reconstruction targets, and this paper adopts that tokenization scheme unchanged.Source: §2 · p. 3
  • Earlier lunar foundation models are either globally distributed at WAC resolution or limited to a few NAC sites and a specific reconstruction task.NASA-IBM LFM combines global coverage, both resolution families, and a reusable representation target.Source: §2 · p. 3
  • Lunar mapping work includes crater detection and IMP segmentation, which test transfer across crater scales and fine, subtle boundaries.The Robbins catalog supports crater mapping, while IMP morphology and debated origins motivate a distinct mapping benchmark.Source: §2 · p. 3
  • Prior lunar terrain methods use stereo, shape-from-shading, or learned reconstruction, whereas this model co-ingests DTM, slope, and aspect as separate modalities.The shared encoder learns their relationships from co-registered data rather than an explicit differential constraint.Source: §2 · pp. 3–4

3 Datasets

SomBench provides geographically partitioned, co-registered lunar data at regional and meter scales, alongside benchmarks for crater detection, IMP segmentation, and polar ice prospectivity. The dataset design aligns auxiliary products to optical anchors and preserves spatially separated evaluation.

3 Datasets

  • SomBench contains approximately two million co-registered tile bundles spanning more than thirty lunar data products and two spatial regimes, with geographic partitioning for leakage-free evaluation.The design supports aligned cross-modal learning, a 100× scale gap, and spatially separated downstream assessment.Source: §3.1 · p. 4

3.1 Pretraining dataset

  • WAC anchors regional tiles at 100 m/px, while NAC anchors meter-scale tiles paired with co-registered 3 m terrain models.The corpus also includes per-tile acquisition metadata, including illumination angles and tile coordinates, available at inference.Source: §3.1 · p. 4
  • An image-anchored pipeline extracts 512×512 tiles and attaches auxiliary products at the optical tile’s geographic bounds.Products are co-registered to the anchor grid; missing observations remain explicitly absent rather than imputed.Source: §3.1 · pp. 4–5, Figure 1 · p. 5
  • Whole map zones or polar caps are assigned to train, validation, or test, keeping overlapping ground observations in the same split across WAC and NAC tracks.The split is approximately 80/10/10, and the test partition is reserved for downstream fine-tuning and evaluation.Source: §3.1 · p. 5

3.2 Benchmarking datasets

  • The dataset section summarizes the corpus composition and benchmark suite, while Table 1 distinguishes dense image-like modalities from scalar context tokens.Dense layers form nine image-like modalities, scalar layers contribute a static-map context modality, and per-tile metadata brings the pretraining total to 11 modalities.Source: §3.2 · p. 5, Table 1 · p. 6, Table 2 · p. 6

Impact Processes

  • Crater benchmarks cover WAC context-scale detection and NAC meter-scale detection, with the NAC set comprising 766 patches and 97,104 crater annotations.The WAC benchmark uses 1,000 visible tiles; NAC annotations are split by site and study area to prevent leakage.Source: Impact Processes · pp. 6–7
  • The IMP benchmark starts from 2,623 polygon annotations, with some corrected for NAC pointing misalignment and polygons larger than 256 m excluded.The authors identify controlled NAC products as a potential source for future benchmark datasets.Source: Impact Processes · p. 7
  • The polar ice benchmark uses eight static layers within 10° of each pole at 240 m/pixel to regress a continuous ice-prospectivity map.The target estimates likely water ice within the upper approximately 1 m of regolith and was originally produced by a knowledge-driven fuzzy overlay.Source: Impact Processes · p. 7

4 Method

NASA-IBM LFM adapts TerraMind’s token-prediction encoder-decoder to lunar modalities, adding explicit acquisition context and joint training across WAC and NAC resolutions. Its modality-specific inputs and patch-size adaptation support varied downstream fine-tuning.

4.1 Modalities and representations

  • NAC- and WAC-centered samples combine dense optical and topographic modalities with sequence-like optical metadata and static-map context, using pixel, VQ-VAE, and text-token branches.The two sample families describe corresponding geographic regions within their available dense modalities.Source: §4.1 · pp. 7–8

4.2 Multimodal pretraining framework

  • Separate Dirichlet draws allocate modality token budgets for inputs and targets; dense tokens are sampled uniformly, sequence inputs span-masked, and discrete targets trained with cross-entropy.Source: §4.2 · pp. 8–9

4.3 Model design considerations

  • Incidence, emission, and phase angles, along with latitude and local solar geometry, enter as context for illumination, shadowing, and surface temperature.Source: §4.3 · p. 10
  • A single training loop mixes NAC (~1 m/pixel) and WAC (~100 m/pixel) batches, while FlexiViT resizes patch embeddings for downstream patch grids without backbone retraining.Each step combines per-modality token losses into one optimization step.Source: §4.3 · p. 10

4.4 Pretraining Setup

  • Pretraining tokenizes data before model training, then trains a ViT-B encoder and Transformer decoder for 150k steps on 16 H100 GPUs with batch size 1,536.The final pretraining run took 1.1k GPU-hours.Source: §4.4 · p. 10

4.5 Finetuning Setup

  • Fine-tuning holds data and evaluation protocols fixed across backbones, compares ImageNet-pretrained and random-init controls, and evaluates rank-16 LoRA adapters for impact and volcanic-history tasks.The pretrained model uses layer-wise learning-rate decay; the ImageNet baselines and from-scratch controls use flat learning rates.Source: §4.5 · p. 11
  • Task heads differ across crater detection, IMP segmentation, and ice-prospectivity regression; WAC crater training uses 50% of its dataset to assess label efficiency.For ice prospectivity, an add-one ablation starts with aspect and slope, then progressively adds modalities in domain-scientist priority order.Source: §4.5 · pp. 11–12

5 Results

Qualitative generation probes indicate cross-modal structure, while downstream results vary by task: WAC crater detection and ice prospectivity favor pretrained NASA-IBM LFM, whereas top NAC crater models are comparable within run variability.

5.1 Multimodal correlations exploration

  • Generated slope and aspect closely tracked DTM-derived targets at both WAC and NAC scales, preserving features such as crater rims and ridge orientations.The paper presents these as qualitative checks, not scientific-grade predictions or a systematic generation evaluation.Source: §5.1 · p. 12
  • In a single WAC example, generated reflectance under different metadata showed illumination aligned with the specified sun direction, although one generation did not match target pixels.This is a single-example demonstration, not a systematic evaluation of geometric fidelity.Source: §5.1 · p. 13
  • Chained generation recovered coarse spatial patterns across modalities, but absolute elevation offsets and latitude-longitude coordinates could drift substantially.The paper attributes these limits to learned local pixel structure without a direct route to a global geodetic reference frame.Source: §5.1 · pp. 13–14

Impact Processes

  • mAP 0.2541±0.0018 with full fine-tuning and 0.2539±0.0014 with LoRA at 50% data exceeded SwinV2-B’s 0.2313±0.0027 using the full WAC training set.At 100% data, LoRA led every reported metric column, including mAP 0.2581±0.0017.Source: Impact Processes · p. 15, Table 4 · p. 16
  • mAP 0.1543±0.0098 for NASA-IBM LFM with LoRA was close to SwinV2-B’s 0.1552±0.0086 on NAC crater detection, with top models within seed variability.The random-init LFM reached mAP 0.1274±0.0151, while the frozen variant reached 0.1295±0.0038.Source: Impact Processes · p. 15, Table 5 · p. 16

Volcanic history

  • IoU1 0.5709±0.0114 and F1 0.7268±0.0093 were the best reported IMP scores, achieved by the frozen NASA-IBM LFM, though top-model differences were within run variability.The random-init control had IoU1 0.3142±0.0746.Source: Volcanic history · p. 17, Table 6 · p. 18

Polar volatiles

  • RMSE 0.0293±0.0013: pretrained NASA-IBM LFM with full fine-tuning led the eight-modality ice prospectivity benchmark, outperforming ImageNet baselines and the random-init control.With three modalities, NASA-IBM LFM reached RMSE 0.0434±0.0006, matching ConvNeXt-B with all eight; with only slope and aspect, both models reached RMSE ≈0.2.Source: Table 7 · p. 18, Polar volatiles · p. 18

6 Conclusion

NASA-IBM LFM combines geometry-conditioned, mixed-resolution pretraining with a reusable checkpoint for lunar tasks. Across four benchmarks it matches or exceeds ImageNet-pretrained baselines, with results varying by task and adaptation strategy.

6 Conclusion

  • One set of weights jointly covers NAC meter-scale and WAC hundred-meter-scale tiles, while FlexiViT serves downstream patch grids without backbone retraining.Acquisition geometry is explicit context, and the model uses a mixed-batch regime for the two tile scales.Source: §6 · pp. 19–20
  • Across four benchmarks, the pretrained model matches or exceeds ImageNet baselines and improves over random initialization; ice-prospectivity RMSE is 0.0293±0.0013 versus 0.0377±0.0004 for the best baseline.The random-init model reaches RMSE 0.0397±0.0004, ahead of five of seven ImageNet-pretrained baselines; pretrained variants retain the lower RMSE.Source: §6 · pp. 20–21
  • LoRA is competitive with full fine-tuning and leads on crater detection, while full fine-tuning retains an edge on the two smallest benchmarks.Freezing the encoder is strongest on IMP but falls below every baseline on crater detection.Source: §6 · p. 21
  • Generation experiments recover terrain derivatives from elevation and illumination-consistent reflectance from geometry, but do not reliably recover absolute elevation offsets or geodetic coordinates.Some static maps are also not reliably recovered; the authors position the model as a downstream representation, not a physical-instrument substitute.Source: §6 · p. 21
  • The contribution of geometry tokenization and mixed-resolution training has not yet been isolated from lunar-domain pretraining, and some benchmarks have limited test samples.The authors note that geodetic-reference-frame recovery remains a gap for applications requiring absolute positioning, though it could in principle be learned during fine-tuning.Source: §6 · p. 21
  • The authors plan targeted ablations, extension to the full modality stack, and evaluation on additional planetary bodies, and release the checkpoint, fine-tuning code, and dataset.The resources are released to support reproducibility and community reuse.Source: §6 · p. 21

Tokenizer training details

The passage points readers to Table 9 for tokenizer training configurations.

Tokenizer training details

  • Table 9 reports training configurations for the domain-specific lunar tokenizers.The supplied passage does not give the configuration values.Source: Tokenizer training details · p. 21

Additional generation results

The section points to further WAC-to-modality generation examples and includes tokenizer training-configuration information.

Additional generation results

  • Table 9 gives per-run training configurations for nine lunar tokenizers, with a shared per-GPU batch size of 32.Source: Table 9 · p. 22
  • Figure 11 presents additional examples of generating all other modalities from WAC.Source: Figure 11 · p. 22

Related papers

Loading 2609.13283v1…