PaperlayerView exact revision on arXiv

Source-linked AI summary

Remote Sensing Sparse-View 3D Gaussian Splatting via Depth Image-Based Rendering

Jiaming Kang, Zhengxia Zou, Zhenwei Shi

arXiv:2609.35612v1cs.CV

TL;DR

Sparse remote-sensing views leave geometry and appearance weakly constrained, especially in unobserved regions. DIBR-GS combines depth-aligned initialization, cross-view appearance features, and progressive pseudo-view supervision; experiments report improved reconstruction quality and a favorable balance with rendering efficiency.

  • Problem

    Sparse remote-sensing observations leave viewpoint regions weakly constrained, with ambiguous geometry, incomplete structures, inconsistent appearance, and insufficient supervision in unobserved areas.

    Source: §I · p. 1, E. Pseudo View Supervision through… · p. 6

  • Method

    DIBR-GS aligns monocular depth with SfM, incorporates cross-view appearance features, and uses progressive DIBR pseudo views to add supervision at unobserved viewpoints.

    Source: §I · p. 2, E. Pseudo View Supervision through… · p. 6

  • Results

    DIBR-GS consistently improves reconstruction quality, producing more complete structures, sharper appearance details, and fewer artifacts in sparse-view remote-sensing scenes.

    Source: §I · p. 2, §V · p. 12

  • Takeaways & Limitations

    The reported results indicate a favorable balance between novel-view rendering quality and computational efficiency for remote-sensing sparse-view synthesis.

    Source: §I · p. 2

Abstract

from arXiv · show

Remote sensing novel view synthesis under sparse observations remains challenging due to insufficient geometric constraints and limited cross-view supervision. Existing Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) methods are prone to overfitting and face challenges of depth ambiguities, missing cross-view information, and insufficient constraints in under-observed regions. To address these challenges, we propose DIBR-GS, a neural Gaussian Splatting framework that exploits Depth Image-Based Rendering (DIBR) to generate pseudo views for cross-view consistency supervision. Specifically, reliable geometric initialization is constructed by aligning monocular depth priors with sparse SfM reconstruction, and cross-view appearance priors are incorporated into neural Gaussian representations to enhance appearance modeling under sparse observations. Furthermore, we introduce a progressive DIBR-based pseudo-view supervision strategy to provide additional geometric and appearance constraints, enabling more complete reconstruction of weakly observed regions. In addition, a height-constrained anchor growth strategy is designed to suppress unreasonable Gaussian expansion. Experiments demonstrate that the proposed method achieves superior performance over existing approaches when training with only 3 input views. Compared with the previous best-performing method, it improves PSNR by 6.83 dB, with relative gains of 14\% in SSIM and 60\% in LPIPS, while maintaining competitive computational efficiency. Our code is available at https://github.com/kanehub/DIBR-GS

I. INTRODUCTION

Sparse remote-sensing views leave geometry, appearance, and unobserved regions weakly constrained. DIBR-GS combines geometric initialization, cross-view appearance features, and progressive pseudo-view supervision to address these gaps.

I. INTRODUCTION

  • NeRF relies on dense per-ray sampling and is computationally costly, whereas 3DGS uses explicit Gaussian primitives and tile-based rasterization for real-time rendering.Both are presented as neural-rendering approaches to novel-view synthesis.Source: §I · p. 1
  • DIBR-GS produces more complete structures and finer details in weakly observed regions while preserving real-time rendering capability.The comparison uses LEVIR-NVS with 3 input views.Source: Figure 1 · p. 1, §I · p. 2
  • Sparse observations can leave large viewpoint regions weakly constrained, causing ambiguous geometry, incomplete structures, and inconsistent appearance.Remote-sensing data may be limited to 3–5 viewpoints because of revisit cycles, flight paths, and occlusions.Source: §I · p. 1
  • Remote-sensing sparse-view methods face unreliable monocular-depth priors, limited joint use of cross-view appearance and geometry, and weak supervision in unobserved regions.These challenges motivate DIBR-GS’s combined geometric, appearance, and pseudo-view constraints.Source: §I · p. 2
  • DIBR-GS aligns monocular depth with SfM, incorporates cross-view geometric and appearance cues, and uses progressive pseudo views plus height-constrained anchor growth.The pseudo views propagate depth priors to unobserved viewpoints for multiview consistency; the height constraint suppresses floating artifacts.Source: §I · p. 2

A. Neural Scene Representations

  • Neural scene representations use neural networks to approximate surface or volumetric functions and combine them with classical rendering for compact 3D representations.The introduction connects this area to implicit, explicit, and hybrid representations.Source: A. Neural Scene Representations · p. 2

B. Novel View Synthesis with Sparse Views

Sparse-view synthesis methods seek to stabilize reconstruction with added priors, but remote-sensing geometry and constrained overhead views remain difficult. DIBR-GS combines depth/SfM initialization, cross-view features, progressive pseudo views, and constrained anchor growth.

B. Novel View Synthesis with Sparse Views

  • Sparse-view approaches add depth supervision or regularization, but monocular-depth scale ambiguity and complex remote-sensing structures challenge their reliability.Prior examples include depth-prior methods and strategies that regularize Gaussian distributions or remove redundant primitives.Source: B. Novel View Synthesis with Sparse… · pp. 2–3

III. METHOD

  • DIBR-GS aligns monocular depth with SfM, fuses views for Gaussian-anchor initialization, integrates cross-view appearance features, and supervises optimization with progressive DIBR pseudo views.A height-constrained anchor-growth strategy also limits unreasonable Gaussian expansion.Source: §III · p. 3, Figure 2 · p. 3

1) 3DGS:

  • 3DGS represents scenes with anisotropic Gaussian primitives whose covariance is parameterized by rotation and scaling, then renders them through differentiable rasterization.The primitives are projected to the image plane and blended in depth order.Source: 1) 3DGS · pp. 3–4

2) Neural Gaussian:

Neural Gaussians are organized around anchors that generate and decode Gaussian attributes. For initialization, monocular depth is aligned with sparse SfM geometry, back-projected, fused, and filtered using near-nadir scene structure.

2) Neural Gaussian:

  • Each Scaffold-GS anchor controls multiple Gaussians, with centers formed from the anchor position plus learned offsets scaled by an anchor factor.Lightweight MLPs decode Gaussian attributes from anchor features, viewing direction, and camera-relative distance; anchors can grow or be pruned during training.Source: 2) Neural Gaussian · p. 4

B. Initialization with Dense Depth Priors

  • Monocular depth is aligned to sparse SfM points with an affine transformation because monocular estimates provide relative depth, then back-projected into world coordinates.The resulting point cloud provides a denser initialization for sparse-view Gaussian optimization.Source: B. Initialization with Dense Depth… · p. 4
  • For near-nadir scenes, fused depth points can form implausible vertically layered surfaces, so the method uses height-distribution tests and neighboring-point overlap checks to remove outliers.The method assumes dominant structures can often be approximated by a single vertical surface distribution.Source: B. Initialization with Dense Depth… · p. 4
  • When redundant surfaces arise during depth fusion, retaining lower-height points provides more stable anchors and mitigates camera-oriented Gaussian drift.The stated rationale is that lower-height points generally correspond to surfaces farther from the camera in near-nadir scenes.Source: B. Initialization with Dense Depth… · p. 4

C. IBR-Based Neural Gaussian Representation

Cross-view reference features add complementary appearance cues to neural Gaussian representations, while adaptive modulation controls their influence when sparse-view references are unreliable.

C. IBR-Based Neural Gaussian Representation

  • Cross-view reference features provide complementary appearance cues to neural Gaussian representations under sparse observations.The features follow an image-based rendering paradigm.Source: C. IBR-Based Neural Gaussian… · p. 5
  • Adaptive modulation controls cross-view feature contributions because occlusions and inaccurate matches can make reference information unreliable.The modulation operates during Gaussian optimization.Source: C. IBR-Based Neural Gaussian… · p. 5
  • A pretrained DINO encoder extracts a local feature pyramid from each input image.The passage begins to denote the pyramid as a set of features Fℓ.Source: 1) IBR Feature Extraction · p. 5

1) IBR Feature Extraction:

The method builds an anchor-level IBR feature by sampling view features at projected anchor locations and aggregating them across views. It combines local texture with global semantic information in a shared feature representation.

1) IBR Feature Extraction:

  • Local texture features and global ViT semantics are projected to a shared dimension and fused to model their complementarity.Source: 1) IBR Feature Extraction · p. 5
  • Each anchor is projected into an input view using its world position and that view’s world-to-camera extrinsics.Source: 1) IBR Feature Extraction · p. 5
  • Bilinear sampling at the projected coordinates yields per-view feature vectors, which are aggregated into the anchor’s final IBR feature.Source: 1) IBR Feature Extraction · p. 5

2) Adaptive Feature Fusion:

Adaptive feature fusion regulates how cross-view IBR information enters neural Gaussian features. Height-constrained anchor growth uses scene height bounds to suppress unreasonable expansion and reduce floating artifacts.

2) Adaptive Feature Fusion:

  • Channel-wise modulation regulates projected IBR features added to the original Gaussian feature, limiting unreliable reference information while preserving useful priors.Source: 2) Adaptive Feature Fusion · p. 5

D. Height-constrained Anchor Growth

E. Pseudo View Supervision through Progressive DIBR

Progressive DIBR synthesizes pseudo views to provide supervision where sparse observations leave novel-view regions uncovered. It combines depth-warped images from multiple input views to expand coverage.

E. Pseudo View Supervision through Progressive DIBR

  • Pseudo views provide complementary supervision to help recover missing structures while maintaining multi-view consistency.The approach addresses insufficient supervision in regions unobserved because of limited viewpoints and occlusions.Source: E. Pseudo View Supervision through… · p. 6
  • Intermediate viewpoints along the input acquisition trajectory are rendered using depth maps completed from a dense point cloud.For each target pixel, depth warping computes its corresponding source-view coordinate before image sampling.Source: E. Pseudo View Supervision through… · p. 6
  • The progressive synthesis fills a target view near-to-far from multiple sources, then depth-warps with padding to complete remaining areas.The strategy integrates complementary visible regions from multiple input views to reduce artifacts from unreliable warping.Source: E. Pseudo View Supervision through… · p. 6

F. Optimization

Training jointly optimizes the neural Gaussian representation and associated modules, with ground-truth and scheduled pseudo-view supervision. Evaluation uses three training views per LEVIR-NVS scene and compares against neural-rendering baselines using four rendering metrics.

F. Optimization

  • Training jointly optimizes anchor Gaussian attributes, the decoder MLP, and IBR feature projection and fusion networks.Ground-truth images guide early structural reconstruction, pseudo views are introduced periodically for consistency and hole removal, then disabled for fine-detail optimization.Source: F. Optimization · p. 6

1) Dataset:

  • LEVIR-NVS evaluation uses 3 of each scene’s 21 images for training and the remaining views for evaluation.The dataset has 16 Google Earth-derived scenes, each with 512×512 images spanning varied land-cover types.Source: 1) Dataset · p. 6

2) Baseline and Metrics:

  • The comparison includes NeRF-, 3DGS-, and remote-sensing-specific methods, evaluated with PSNR, SSIM, LPIPS, and AVGE.The study follows the same experimental settings and training protocols for compared methods.Source: 2) Baseline and Metrics · pp. 6–7

B. Implementation Details

Implementation constructs a dense point-cloud initialization by aligning monocular depth with SfM, builds cross-view reference features, and constrains neural Gaussian growth with a height map.

B. Implementation Details

  • Monocular depth is affine-aligned to sparse COLMAP point clouds, refined to remove overlap, voxel-downsampled, and merged with SfM points for Gaussian initialization.Depth Anythingv2 supplies the depth maps; the overlap filter partitions the world-coordinate XY plane into grids and tests height distributions.Source: 1) Architecture Details · p. 7
  • Reference features sum projected ResNet-50 layer maps and spatially replicated ViT-B/8 global and local tokens in a 32-dimensional feature space.Both feature extractors are frozen and pretrained with a DINO objective.Source: 1) Architecture Details · p. 7
  • A dense-point-cloud height map constrains anchor growth, using each XY grid’s 95th-percentile height with a relaxed boundary.The height constraint is applied only at the lowest resolution to avoid disrupting fine structural details; pseudo-view poses are sampled from the circular acquisition trajectory.Source: 1) Architecture Details · pp. 7–8

2) Training Details:

DIBR-GS outperforms the competing approaches on LEVIR-NVS and produces stable rendering across varied scenes. Ablations indicate complementary roles for initialization, appearance features, height-constrained growth, and pseudo-view supervision.

2) Training Details:

  • Training runs for 30,000 iterations, with densification every 100 iterations from iteration 1,500 to 15,000 and pseudo-view sampling every 50 iterations from iteration 3,000 to 20,000.Experiments use one Nvidia GeForce RTX 4090 GPU.Source: 2) Training Details · p. 8

C. Comparison with Other Methods

  • DIBR-GS achieves superior reconstruction quality over all competing approaches, with stable, high-quality rendering across scenes of different types and complexities.Per-scene experimental results are reported in Table II.Source: C. Comparison with Other Methods · p. 8, Table II · p. 9
  • Visual comparisons show sharper structures and more recovered texture, while reducing floating artifacts and unreasonable geometric expansion.Compared methods exhibit blurred details, ghosting, shape distortion, or missing structures in challenging regions.Source: C. Comparison with Other Methods · p. 8

1) Ablation of Model Architecture:

  • Removing any single component degrades performance; pseudo-view removal particularly increases holes and structural collapse in unobserved regions.Dense-depth initialization supports geometric stability, IBR fusion supports texture and cross-view appearance, and height-constrained growth reduces floating artifacts and excessive expansion.Source: 1) Ablation of Model Architecture · p. 9, Table III · p. 9

2) Ablation of Initialization:

The initialization and network ablations examine depth alignment, anchor redundancy, and feature capacity, identifying settings that balance reconstruction performance with computational cost.

2) Ablation of Initialization:

  • Reciprocal disparity mapping best matches the original depth distribution, while local point-cloud overlap removal mitigates depth conflicts without damaging foreground depth.Fused points provide more scale-adjustment data but add noise; Reproject alleviates background-depth underestimation while distorting foreground depth.Source: 2) Ablation of Initialization · pp. 9–10, Figure 6 · p. 10
  • Above 100k initial points, final anchor count nearly saturates; beyond 280k it decreases, while downsampling improves efficiency without compromising reconstruction quality.The reported trend shows performance gains become much smaller relative to added computational cost beyond an elbow point.Source: Figure 7 · p. 10, Table V · p. 10, 2) Ablation of Initialization · p. 11
  • Finer anchor voxels improve reconstruction performance but increase anchor count and computational overhead.The comparison is conducted on Building#1 with the number of initial points fixed.Source: Figure 8 · p. 10, Table VI · p. 10, 2) Ablation of Initialization · p. 11

3) Ablation of Network:

  • With anchor feature dimension Cf = 32, Cr = 32 gives the best performance; increasing IBR dimension to 256 brings only minor variations, while Cf = 64 adds marginal gains over 32.The final configuration is Cf = 32 and Cr = 32, selected for performance and computational efficiency.Source: Table VII · p. 11, 3) Ablation of Network · p. 11
  • Removing the modulation network, freezing feature projection, or excluding ResNet or ViT features degrades performance.The reported ablation indicates local texture cues and global semantic information are complementary for cross-view feature modeling.Source: Table VIII · p. 11, 3) Ablation of Network · p. 11

E. Analysis of DIBR-based Pseudo-view Supervision

Progressive DIBR synthesizes pseudo views by warping and combining source views, adding supervision for regions weakly covered by sparse observations. The reported final rendering reaches 27.86 PSNR despite lower pseudo-view fidelity, and the paper notes that optimization remains per-scene.

E. Analysis of DIBR-based Pseudo-view Supervision

  • Intermediate target viewpoints are sampled along the training-camera trajectory; depth-based warping projects source views, which are integrated from nearer to farther before padding uncovered regions.The example shows that projections from multiple source views expand target-view coverage.Source: E. Analysis of DIBR-based Pseudo-view… · pp. 11–12, Figure 9 · p. 12
  • 27.86 PSNR is reported for the final rendered image despite pseudo-view PSNR of 15.82; removing pseudo-view supervision produces noticeable holes.The pseudo views are described as approximate geometric and appearance constraints, not pixel-perfect targets.Source: E. Analysis of DIBR-based Pseudo-view… · p. 12

V. CONCLUSION

  • DIBR-GS improves sparse-view reconstruction quality, but still relies on per-scene optimization; the conclusion points to feed-forward methods as future work.The conclusion characterizes the framework as a solution for remote-sensing reconstruction and scene understanding when dense observations are unavailable.Source: §V · p. 12

Related papers

Loading 2609.35612v1…