PaperlayerView exact revision on arXiv

Source-linked AI summary

MoonGS: High-quality Representation of the Lunar Surface via Gaussian Splatting Using Robust Depth Features from Image Pairs

Yun Jiang, Bo Zheng, Yingying Zhang, Xueming Xiao, Tao Hu, Hutao Cui, Zhiguo Meng, Ke Gao, Yang Gao, Meibao Yao

arXiv:2610.07110v1cs.CV

TL;DR

Sparse lunar images make high-fidelity, real-time novel-view synthesis difficult because overlap, texture, and observation volume are limited. MoonGS predicts Gaussian scenes from paired images using robust depth features, semantic priors, and resampling. It reports higher PSNR than comparison methods on both LuSNAR and MoonBlender.

  • Problem

    Lunar reconstruction from sparse rover images is challenged by limited viewpoint overlap, weak background textures, and mission constraints on observations.

    Source: §I · pp. 1–2

  • Method

    MoonGS predicts a Gaussian scene from sparse lunar images using robust vision-foundation-model depth features, semantic priors, a semantic ranking loss, and heuristic resampling.

    Source: §I · p. 2

  • Results

    23.80 dB PSNR versus the other methods’ 18.89 dB high was reported on LuSNAR; MoonGS reached 26.08 dB versus pixelSplat’s 25.26 dB on MoonBlender.

    Source: C. Comparative Results · p. 8

  • Takeaways & Limitations

    The paper frames high-fidelity lunar novel views as supporting rover terrain understanding, visual localization, and real-time decision-making.

    Source: §I · p. 1

Abstract

from arXiv · show

High-quality 3D reconstruction of lunar terrain from sparse rover images is indispensable for autonomous lunar exploration, but remains challenging because viewpoint overlap is insufficient, surface textures are weak, and data volume is limited. We propose MoonGS, the first feed-forward 3D Gaussian Splatting framework tailored to lunar scenes. Given only two input images, MoonGS predicts pixel-aligned Gaussian primitives in a single forward pass and renders photorealistic novel views without any per-scene optimization. MoonGS (i) adopts an adaptable backbone design that seamlessly integrates advanced vision foundation models to extract robust depth features; (ii) integrates semantic priors in two manners: merging semantic cues with visual features to refine Gaussian parameter estimation, and adopting a semantic ranking loss that regularizes background depth; and (iii) employs an entropy-guided heuristic resampling strategy to augment sparse observations by selecting the most informative distant viewpoints with negligible overhead. Experiments on the LuSNAR benchmark and our synthetic weak-texture MoonBlender dataset show that MoonGS surpasses state-of-the-art feed-forward NeRF/3DGS baselines by +4.9 dB PSNR, +0.29 SSIM, and 40\% lower LPIPS while maintaining sub-second inference. Furthermore, we validate the broad applicability of our framework by demonstrating that it effectively leverages state-of-the-art backbones, including VGGT, to significantly boost performance. Qualitative evaluations on Chang'e mission imagery also show the best visual quality among compared methods, indicating robustness on real lunar data. The source code and dataset are publicly available at https://github.com/InRobots/MoonBlender.

I. INTRODUCTION

Sparse lunar observations challenge high-fidelity, real-time novel-view synthesis, but such views can support rover terrain understanding, localization, and decision-making. MoonGS addresses this with feed-forward Gaussian reconstruction using robust depth features, semantic information, and heuristic resampling.

I. INTRODUCTION

  • High-fidelity novel views are framed as supporting rover terrain understanding, precise visual localization, and real-time decision-making.The introduction illustrates the novel-view comparison in Fig. 1.Source: §I · p. 1, Figure 1 · p. 1
  • Sparse lunar reconstruction is difficult because low viewpoint overlap undermines cross-view matching, weak background textures complicate depth estimation, and mission constraints limit observations.Conventional 3DGS also requires per-scene training and densely captured images, limiting its suitability for sparse lunar missions.Source: §I · pp. 1–2
  • MoonGS uses robust vision-foundation-model depth features, semantic priors for Gaussian prediction, a semantic ranking loss for weak-texture depth, and heuristic resampling to augment sparse observations.The introduction also presents MoonBlender, a synthetic dataset designed to evaluate reconstruction under weakly textured conditions.Source: §I · p. 2
  • MoonGS is a feed-forward 3DGS framework designed to synthesize high-fidelity novel views from sparse lunar images in real time.It represents scenes with Gaussian primitives and predicts them in a forward pass.Source: §I · p. 2

II. RELATED WORK

Prior celestial reconstruction and feed-forward rendering methods face limits in data sparsity, speed, and planetary-scene overlap. MoonGS combines foundation-model depth features, lunar semantics, and resampling in a feed-forward Gaussian reconstruction pipeline.

A. Celestial Surface Reconstruction Based on Radiance Fields

  • Celestial novel-view methods progressed from light fields, SfM, and MVS to neural rendering, but geometry errors, data sparsity, and limited real-time performance remain concerns for lunar applications.The paper positions sparse lunar image pairs and a specialized architecture as its response to those application needs.Source: A. Celestial Surface Reconstruction… · p. 2

B. Feed-forward Radiance Fields

  • Feed-forward NeRF methods infer from sparse views but retain ray-sampling costs, while feed-forward 3DGS methods predict explicit Gaussian scenes efficiently.Cost-volume methods improve geometry, yet end-to-end MVS approaches remain challenged by the limited overlap in planetary data; MoonGS instead uses foundation-model depth features, lunar semantics, and heuristic resampling.Source: B. Feed-forward Radiance Fields · p. 3

C. Key Components of Feed-forward 3DGS

  • Depth prediction is central to feed-forward 3DGS because it determines Gaussian positions; DUSt3R and VGGT pointmap features are presented as improving depth consistency and accuracy.The cited discussion contrasts these pretrained foundation models with approaches relying on image encodings and cross-attention for cross-view features.Source: C. Key Components of Feed-forward 3DGS · p. 3

III. METHODS

  • MoonGS infers Gaussian parameters from paired images, camera parameters, and semantic images in a feed-forward procedure, then renders unobserved viewpoints.Its method overview assigns robust depth extraction, semantic integration, and resampling to successive method components.Source: §III · pp. 3–4

A. Motivations and Preliminaries

Sparse rover imagery can produce artifacts or reconstruction failures where epipolar constraints are invalid, observations are insufficient, or sky depth is ambiguous. The section introduces Gaussian primitives as an explicit scene representation and notes their efficient splat rendering.

A. Motivations and Preliminaries

  • Forward-moving rover imagery can leave non-overlapping regions, distant areas with too few observations, and low-texture sky regions with ambiguous depth.The paper identifies these conditions as limitations of sparse-view reconstruction approaches including pixelSplat and MVSplat.Source: A. Motivations and Preliminaries · p. 4
  • A Gaussian primitive is defined by its position, covariance, color, and opacity, with covariance parameterized by scale and rotation.The representation uses anisotropic Gaussian primitives to model entire scenes.Source: A. Motivations and Preliminaries · p. 4
  • 3DGS accumulates Gaussian opacity for rendering and uses parallelizable tile-based rendering rather than NeRF-style ray sampling.The reconstruction goal is to estimate the full parameter set for each Gaussian in the scene.Source: A. Motivations and Preliminaries · p. 4

B. Problem Definition

MoonGS predicts scene-wide, pixel-aligned Gaussian parameters from paired images and camera information, then uses them to render unobserved viewpoints. Its depth-feature pipeline combines cross-view features from a vision foundation model with cost-volume construction and feature refinement.

B. Problem Definition

  • MoonGS maps paired images and camera parameters to pixel-aligned Gaussian positions, opacities, covariances, and colors for rendering from unobserved viewpoints.Gaussian position prediction depends on depth prediction and unprojection using camera parameters.Source: B. Problem Definition · p. 4
  • Insufficient viewpoint overlap, weakly textured backgrounds, and limited distant observations create distinct challenges for lunar reconstruction.The figure associates these conditions with invalid epipolar constraints, error-prone constraints, and difficult high-fidelity texture prediction, respectively.Source: Figure 4 · p. 4

C. Robust Depth Features and Cost Volume

  • DUSt3R’s shared-encoder branches decode paired images jointly through self- and cross-attention to produce cross-view depth features.The same weight-sharing ViT encoder processes each image before the decoder exchanges view-related information.Source: C. Robust Depth Features and Cost Volume · p. 5
  • MoonGS fuses decoder features from the last two scales and refines them with a lightweight 2D U-Net residual before downstream prediction.The DUSt3R parameters are frozen; the refined features combine original features with the network’s residual output.Source: C. Robust Depth Features and Cost Volume · p. 5
  • A MVSplat-style cost volume provides smoother, unified-scale depth and converts DUSt3R’s implicit camera representation for Gaussian prediction.It samples depth candidates and warps features between views; predicted depth is unprojected to determine Gaussian positions.Source: C. Robust Depth Features and Cost Volume · p. 5

D. Integrating Semantics for Enhanced Performance

MoonGS uses semantic information both to condition Gaussian-parameter prediction and to regularize depth ordering in low-texture regions. The semantic ranking loss avoids fixing background depth to an absolute far distance.

Semantic fusion

  • Concatenating one-hot semantic labels with visual features conditions depth-map and Gaussian-parameter prediction on category information.The paper motivates this prior by category-specific texture characteristics, while noting that some categories, such as regolith and impact craters, may have similar textures.Source: D. Integrating Semantics for Enhanced… · p. 5
  • When semantics are unreliable or absent, MoonGS gates out semantic fusion and uses appearance- and geometry-based features.Source: D. Integrating Semantics for Enhanced… · p. 6

Semantic ranking loss

  • The semantic ranking loss regularizes low-texture depth through known nearer–farther relations rather than absolute metric depth.It can apply to any region pair with a known depth ordering, and is disabled when semantics are unavailable or low quality.Source: D. Integrating Semantics for Enhanced… · p. 6, Figure 5 · p. 6

Gaussian coverage statistics

  • Fig. 6 counts valid pixels occupied by each Gaussian, then sums those counts for the Gaussians covering each pixel in parallel on the GPU.Source: Figure 6 · p. 6

E. Heuristic Resampling Strategy

MoonGS uses an entropy-guided, self-supervised heuristic to prioritize candidate views for resampling. The score is computed from Gaussian coverage statistics and selects locations with the highest entropy.

E. Heuristic Resampling Strategy

  • Sparse observations and predominantly forward rover motion leave distant or single-view surfaces difficult to reconstruct in detail.The paper motivates resampling as a response to lunar depth range and mission constraints on viewpoint number and diversity.Source: E. Heuristic Resampling Strategy · p. 6
  • The view score is the entropy of normalized per-pixel totals of the footprint sizes of Gaussians covering each pixel.The statistics are computed for candidate views after splatting.Source: E. Heuristic Resampling Strategy · p. 6
  • High-entropy candidate locations are prioritized because dispersed coverage and overlapping footprints signal under-coverage or ambiguity in sparse two-view scenes.The procedure is self-supervised, needs no ground-truth images, and adds negligible computational overhead through parallel implementation.Source: E. Heuristic Resampling Strategy · p. 6

IV. EXPERIMENTS

The experiments compare MoonGS with baseline methods on lunar datasets and use ablations to assess framework components. They also examine extension to advanced architectures such as VGGT.

IV. EXPERIMENTS

  • The experiments compare lunar-dataset performance with baselines, ablate framework components, and examine extension to architectures such as VGGT.Source: §IV · p. 7

A. Datasets

The evaluation uses LuSNAR and a purpose-built weak-texture MoonBlender dataset, with additional qualitative checks on Chang’e imagery. The datasets provide semantic labels and camera information for sparse-view lunar reconstruction.

A. Datasets

  • LuSNAR provides multiple scenes, camera poses, and five semantic categories, including sky, regolith, rocks, craters, and mountains.Its sky is a weak-texture area with a large pixel share; its rocks are relatively few and irregular.Source: A. Datasets · p. 7
  • MoonBlender extends LuSNAR to test weak-texture, sparse-view reconstruction with more complex rock distributions and a larger, weaker-texture sky background.It contains 1,500 1200×1200 images rendered along five traversable routes, with varied Sun azimuth and elevation and automatically obtained camera and semantic metadata.Source: A. Datasets · p. 7
  • Among 96 randomly sampled Chang’e 4 TCAM images, 53 contained sky, with a mean sky-background pixel ratio of 42.11% among those 53.The figure’s reported Chang’e ratios are computed over the sky-containing subset.Source: A. Datasets · p. 7, Figure 7 · p. 7
  • The study uses the last four of LuSNAR’s nine scenes for training and validation, with MoonBlender used for auxiliary validation and Chang’e imagery for qualitative real-data evaluation.The evaluated image resolution is 256×256; Chang’e view pairs were selected after grouping by acquisition time and estimating poses with COLMAP.Source: A. Datasets · p. 7

B. Experimental Settings

MoonGS is evaluated against feed-forward NeRF and 3DGS methods on LuSNAR and MoonBlender, with additional qualitative tests on Chang’e imagery. The comparisons report higher image quality, resource use, resampling, depth behavior, and compatibility with VGGT.

B. Experimental Settings

  • The comparisons include feed-forward NeRF and 3DGS methods, with PSNR, SSIM, and LPIPS used for image quality and additional measurements of model resources.The training objective combines MSE, LPIPS weighted 0.05, and ranking loss weighted 0.01; models were evaluated at batch size one on a single Q6000 GPU.Source: B. Experimental Settings · p. 8

C. Comparative Results

  • With baselines retrained on both datasets, MoonGS achieved 23.80 dB PSNR on LuSNAR versus the other methods’ 18.89 dB maximum, and 26.08 dB on MoonBlender versus pixelSplat’s 25.26 dB.On LuSNAR, MoonGS also outperformed the baselines in SSIM and LPIPS.Source: C. Comparative Results · p. 8
  • MoonGS produced more reliable, smoother depth maps than the compared methods, while pixelSplat showed noisy low-texture depths and MVSplat showed projection errors on MoonBlender.The semantic depth-ranking loss further improved optimization of weakly textured sky regions.Source: C. Comparative Results · pp. 9–10, Figure 11 · p. 11
  • On Chang’e imagery across three illumination conditions, MoonGS had no reconstruction failures and stayed closer to the references, while pixelSplat and MVSplat showed lighting-related artifacts.When input illumination differed markedly, a visible boundary could appear, although the reconstructions remained closer to the references and preserved fine detail.Source: C. Comparative Results · p. 10
  • MoonGS outperformed MVSplat, the closest competitor in inference speed, by at least 0.1 seconds, while adding around 6M trainable parameters over MVSplat.The authors attribute the performance gains primarily to more effective feature representations rather than model size alone; the DUSt3R backbone remained frozen.Source: C. Comparative Results · p. 11
  • With VGGT as its frozen backbone, MoonGS improved performance with 50M trainable parameters and outperformed baselines in the reported LuSNAR comparisons.The qualitative comparison also reports better visual quality than the baseline without auxiliary modules.Source: Table II · p. 11, C. Comparative Results · p. 11, Figure 12 · p. 11
  • Heuristic resampling produced richer detail and fewer artifacts, with improvements more pronounced near the second reference image.The self-supervised selection targets distant observations and does not use ground truth.Source: Figure 10 · p. 10, C. Comparative Results · p. 11

D. Ablation Results

The ablations examine feature refinement, semantic fusion, ranking loss, and heuristic resampling. They report improvements in rendering quality and depth estimation, including when components are applied to pixelSplat.

Multi-level feature fusion and U-Net

Semantic fusion

  • Semantic fusion improved rendering quality by approximately 0.2 dB on MoonBlender and about 0.8 dB on LuSNAR.Semantic features separated categories that otherwise became entangled; when masks were unreliable, the fusion branch was gated off and the model still surpassed baselines on both datasets in PSNR, SSIM, and LPIPS.Source: D. Ablation Results · p. 13, Figure 15 · p. 13

Semantic ranking loss

  • Ranking loss regularized weak-texture depth, resolving abnormal sky depth and yielding more reliable sky depth when integrated into pixelSplat.The loss regularizes depth through ordinal relations; the figure shows its effect on weakly textured background regions.Source: Figure 13 · p. 12, D. Ablation Results · p. 13

Heuristic resampling strategy

  • 24.09 dB PSNR was the highest score for the heuristic resampling strategy on the LuSNAR video sequence, outperforming fixed-interval resampling.The strategy uses little prior information, evaluates positions in parallel with rendering, and also improves pixelSplat and other models.Source: D. Ablation Results · pp. 13–14, Table IV · p. 14

V. DISCUSSIONS

The discussion identifies boundaries involving semantic inputs, view count, device size, backbone training, and camera poses. It also describes planned extensions to address these constraints.

Semantic inputs

  • Several pipeline modules use semantics; a fallback preserves stability when semantics are unavailable.The authors plan to reduce this dependence with more readily obtained priors, including edge and regionness cues, uncertainty-aware pseudo-labels, and surface normals.Source: §V · p. 14

View count and map scalability

  • MoonGS focuses on pair-based feed-forward synthesis, while larger maps can be formed by fusing multiple inference outputs.Per-pass memory depends on image resolution and per-primitive state; hierarchical tiled representations and spatially adaptive splatting are proposed for future fused-map efficiency and scalability.Source: §V · p. 14

Constrained devices

  • Despite fast inference, the network is not yet lightweight enough for constrained devices.Model distillation, pruning, and quantization are proposed to reduce model size while preserving accuracy.Source: §V · p. 14

Backbone training

  • The default DUSt3R backbone remains fixed during training, a choice made to improve stability and data efficiency with limited lunar data.With larger and more diverse datasets, the authors plan to partially unfreeze and fine-tune selected layers.Source: §V · p. 14

Foundation-model integration

  • The discussion reports VGGT integration and parameter efficiency with a frozen foundation model, while integration of next-generation models remains future work.Source: §V · p. 14

Camera poses

  • The current pipeline assumes known relative camera poses as input; pose-free variants are future work.Source: §V · p. 14

VI. CONCLUSION

The conclusion presents MoonGS as a lunar-focused feed-forward 3DGS method and reports higher-quality reconstructions on two lunar datasets than state-of-the-art methods. It also describes MoonBlender, real Chang’e imagery results, and the method’s relevance to rover perception.

Method and findings

  • MoonGS combines foundation-model depth features, semantic priors, ranking-loss depth regularization, and heuristic resampling for lunar reconstruction from sparse views.The conclusion reports improved results in non-overlapping and weakly textured regions.Source: §VI · p. 14

Benchmark results

  • MoonGS achieved higher-quality reconstructions than state-of-the-art methods, on two lunar datasets, including in non-overlapping or weakly textured regions.Source: §VI · p. 14

Evaluation scope

  • MoonBlender was introduced to evaluate reconstruction under weakly textured conditions, and Chang’e imagery showed higher-fidelity reconstructions and robustness under varying illumination.Source: §VI · p. 14

Implication

  • The conclusion states that synthesized novel viewpoints improve rover perception and enable more efficient and safer lunar exploration.Source: §VI · p. 14

Related papers

Loading 2610.07110v1…