PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing the Unseen: Semantic-in-Gaussian for Sparse-View 3D Generalization

Zeyang Bai, Yunpeng Wang, Yunbiao Wang, Jun Xiao

arXiv:2608.22740v1cs.CV

TL;DR

Sparse-view G-3DGS remains constrained by pixel-aligned Gaussian estimation in partially observed and occluded regions. SeeU uses CEA-conditioned residual refinement in Gaussian space through a Conditional Gaussian Transformer. Across multiple benchmarks, it improves rendering quality and structural completeness, with a reported +2.32 dB PSNR over HiSplat on RealEstate10K extrapolation.

  • Problem

    Existing G-3DGS frameworks rely on pixel-aligned Gaussian estimation, which struggles in partially observed or occluded regions and can produce incomplete surfaces or structural collapse.

  • Method

    SeeU aggregates multi-view semantic and geometric cues with CEA embeddings and uses them to condition a Gaussian Transformer that predicts residual updates to coarse Gaussians.

  • Results

    +2.32 dB PSNR is achieved by SeeU over HiSplat on RealEstate10K extrapolation, while experiments across multiple settings report consistently improved rendering quality.

  • Takeaways & Limitations

    Semantic-conditioned refinement in Gaussian space improves reconstruction fidelity, particularly in occluded and unseen regions, while retaining feed-forward inference.

Abstract

from arXiv · show

Generalizable 3D Gaussian Splatting (G-3DGS) has emerged as a promising approach for novel view synthesis undersparse-view settings. However, existing frameworks remain restricted by pixel-aligned Gaussian estimation, whichstruggles in partially observed or occluded regions and often leads to incomplete surfaces or structural collapse. Toaddress these challenges, we propose SeeU (Seeing the Unseen), a novel G-3DGS framework. We frame its core design asSemantic-in-Gaussian: semantic-conditioned refinement in Gaussian space. Specifically, we introduce a Cross-viewEntropy-Aware (CEA) module that aggregates multi-view semantic and geometric cues into compact embeddings. Theseembeddings guide the Conditional Gaussian Transformer, which applies residual updates to coarse Gaussians, helpingrecover under-constrained regions of partially observed structures while preserving surface consistency. Comprehensiveexperiments on multiple benchmarks demonstrate that SeeU consistently improves rendering quality and structuralcompleteness while retaining efficient feed-forward inference. Especially under challenging extrapolation settings,SeeU achieves an average improvement of 2.44 dB in PSNR compared to recent SOTA G-3DGS methods.

1 Introduction

Sparse-view G-3DGS methods are limited by pixel-aligned estimation, which struggles in occluded and weakly observed regions. SeeU addresses this with semantic-conditioned Gaussian refinement using CEA embeddings and a Conditional Gaussian Transformer.

  • G-3DGS enables rapid feed-forward reconstruction from only a few input views, avoiding per-scene optimization and costly dense-view acquisition.
  • Pixel-aligned depth estimation propagates errors into Gaussian geometry and can produce black holes or collapsed structures under sparse-view occlusions and weak textures.
  • SeeU shifts reconstruction from direct pixel-space estimation to semantic-guided refinement in Gaussian space.
  • CEA aggregates multi-view semantic cues with depth-distribution entropy to emphasize weakly constrained regions.
  • +2.32 dB PSNR is achieved by SeeU over HiSplat on RealEstate10K extrapolation.

2 Related Work

Sparse-view generalizable 3DGS targets feed-forward novel-view synthesis but remains challenged by pixel-aligned estimation in unseen regions. SeeU addresses this with semantic-guided Gaussian refinement that compensates for under-constrained areas while preserving structural alignment.

  • Novel view synthesis renders photo-realistic images from limited input views, while NeRF-based methods can be slow, memory-intensive, and suboptimal for sparse viewpoints.
  • Sparse-view generalizable 3DGS methods predict Gaussian representations for unseen scenes without per-scene re-optimization, using mechanisms such as epipolar transformers, plane sweeping, and depth-aware matching.
  • SeeU fuses per-view class embeddings through the Cross-view Entropy-Aware module and uses them to guide residual refinement of latent Gaussians with a Conditional Gaussian Transformer.
  • Pixel-aligned Gaussian estimation struggles to recover fine details and resolve ambiguities in unseen regions.
  • Unlike conditional 3D generation methods primarily targeting content creation, SeeU uses semantic compensation for under-constrained regions while preserving structural alignment with input views.

3 Methodology

SeeU reconstructs scenes by initializing coarse pixel-aligned Gaussians, conditioning them with cross-view semantic and uncertainty cues, and applying residual refinement in Gaussian space. Its transformer-based design preserves observed-view geometry while targeting weakly constrained regions.

  • Gaussian Initialization: SeeU initializes latent 3D Gaussians from sparse multi-view images using aggregated features and coarse scene geometry.The representation retains pixel-to-Gaussian correspondence and uses camera parameters to form geometrically consistent preliminary centers.
  • Gaussian Initialization: A multi-view encoder extracts reduced-resolution features with cross-view attention, preserving interactions while shortening the refinement sequence.The resulting features initialize latent Gaussian parameters.
  • Coarse Matching: Plane-swept cost volumes compare warped features across depth candidates to estimate coarse depth and matching uncertainty.Features from other views are warped using camera parameters, and softmax converts the cost volume into per-view depth distributions.
  • Cross-view Entropy-Aware Embedding: The CEA module combines semantic features with matching entropy to emphasize weakly constrained regions and aggregate complementary evidence across views.Entropy is derived from depth posteriors, while cross-attention and Perceiver-style aggregation produce a compact conditioning embedding.
  • Evaluation: SeeU is evaluated on interpolated NVS across RealEstate10K and ACID using three novel views rendered from two reference viewpoints.The reported comparison averages performance across scenes and notes that 3DGS methods render at approximately 500 FPS.
  • Residual Refinement: A DiT-based Conditional Gaussian Transformer flattens Gaussian parameters into tokens and predicts CEA-conditioned residual updates.The residual formulation corrects weakly constrained regions while preserving geometry anchored by the input views.

4 Experiments and Discussions

SeeU is evaluated across within-dataset, extrapolated, and zero-shot settings, where it improves rendering quality and preserves geometry in weakly constrained regions. The framework also retains efficient inference and benefits from Gaussian refinement.

  • Experimental Settings: SeeU is evaluated on RealEstate10K and ACID using two context images and three target views for interpolated and extrapolated NVS.The datasets contain 67,477/7,289 and 11,075/1,972 training/test scenes, respectively.
  • Within-Dataset Results: SeeU improves performance on both RealEstate10K and ACID while preserving fine structures and reducing occluded-region artifacts.These gains are attributed to CEA-conditioned residual refinement.
  • Efficiency: 0.089 s per frame is the full framework’s inference time, substantially faster than HiSplat at 0.510 s.The runtime remains competitive with lightweight baselines.
  • Extrapolated NVS: 2.32 dB PSNR over HiSplat is achieved by SeeU on extrapolated RealEstate10K NVS, alongside a 20% LPIPS reduction.Extrapolated targets lie outside the reference-view range, and SeeU better preserves boundary structures in weakly constrained regions.
  • Zero-Shot Cross-Dataset Evaluation: SeeU improves rendering fidelity and geometric accuracy simultaneously under zero-shot cross-dataset transfer to DTU.It obtains Depth RMSE 0.841 and Overall Chamfer 6.75, while producing smaller residuals and better-preserved boundaries.
  • Ablation: Removing Gaussian refinement causes structural distortions and incomplete geometry, whereas the full model produces sharper, more complete reconstructions.Replacing the Conditional Gaussian Transformer with a 2D UNet produces blurry edges.

4.3 Ablation Studies

The ablations identify Gaussian refinement and CEA conditioning as important components, while training with a moderate extrapolation range provides the strongest balance between challenge and visual support.

  • Gaussian Refinement: Removing Gaussian refinement causes collapsed structures and missing surfaces, while replacing the Conditional Gaussian Transformer with a 2D UNet blurs edges.The comparisons use models trained on RealEstate10K with two input views.
  • Conditional Embeddings: CEA improves PSNR by +1.02 dB and reduces LPIPS by ∼8.8% relative to BLIP under identical conditioning interfaces and training settings.The encoders are frozen and projected into the same 768-dimensional token space.
  • Conditional Embeddings: CEA produces sharper edges and more complete geometry than DINOv3, CLIP, and BLIP-2 embeddings.DINOv3 overlooks boundaries, while CLIP and BLIP struggle with fine structures such as sofa contours.
  • Extrapolation Training Range: 45-frame extrapolation training outperforms both no extrapolation and the 90-frame setting.Very large offsets increase ambiguity, whereas a moderate range retains sufficient visual evidence to associate semantics with geometry.

5 Conclusion

SeeU introduces semantic-conditioned refinement in Gaussian space for sparse-view novel view synthesis. Its CEA-conditioned Conditional Gaussian Transformer improves reconstruction fidelity, especially in occluded and unseen regions.

  • Conclusion: SeeU applies CEA-conditioned residual updates directly to coarse Gaussians, improving geometry in partially observed or uncertain scenes.The approach combines semantic guidance with Gaussian-space refinement rather than relying solely on pixel-aligned estimation.
  • Conclusion: SeeU consistently enhances reconstruction fidelity across multiple datasets, particularly in occluded and unseen regions.The conclusion identifies semantic-conditioned refinement in Gaussian space as a promising direction for generalizable 3D reconstruction.

A Implementation Details

SeeU uses a lightweight feed-forward implementation with two sparse input views, frozen DINOv3 features, and a four-layer Conditional Gaussian Transformer.

  • Implementation Details: Input images are resized to 256×256, two sparse views are used, and the latent resolution is 64×64 with downsampling factor s = 4.The semantic encoder is a frozen ViT-Base pretrained with DINOv3.
  • Implementation Details: The Conditional Gaussian Transformer follows TinyDiT with four layers, hidden size 256, four attention heads, and MLP ratio 2.

B.1 Zero-Shot Evaluation on ACID

SeeU generalizes from RealEstate10K to ACID without fine-tuning, achieving the best PSNR and SSIM while HiSplat retains a marginal LPIPS advantage.

  • SeeU achieves the best PSNR and SSIM when models trained on RealEstate10K are evaluated on ACID without fine-tuning.
  • 0.04 dB and 0.003 are SeeU’s improvements over HiSplat in PSNR and SSIM, respectively.
  • HiSplat retains a marginal 0.001 advantage over SeeU in LPIPS under this domain shift.

B.2 Generalization to More Input Views

SeeU is evaluated with more sparse input views than used during training by transferring a 2-view RealEstate10K model to 3-view DTU inputs.

  • SeeU is evaluated zero-shot on DTU using 3-view inputs after training with 2-view inputs on RealEstate10K.
  • The method demonstrates robust performance when evaluated with more input views.

B.3 More Visual Comparisons

Qualitative comparisons across RealEstate10K and ACID highlight SeeU’s sharper, more structurally consistent novel views under challenging regions and cross-dataset evaluation.

  • On RealEstate10K, PixelSplat and MVSplat frequently produce severe blurring and geometric distortions, while HiSplat remains sharper but misses boundaries and occluded regions.
  • Figure 8 presents zero-shot qualitative comparisons on ACID after training all models on RealEstate10K without fine-tuning.
  • Table 8 reports quantitative comparisons for 3-view cross-dataset generalization.
  • SeeU consistently generates sharper novel views that preserve furniture outlines and textile patterns.
  • Figure 9 highlights challenging regions with red boxes, including artifacts, texture loss, object contours, furniture edges, and patterned bedspreads.
Loading 2608.22740v1…