PaperlayerView exact revision on arXiv

Source-linked AI summary

PePESeg3D: Perception Prior Enhances Multi-Scale Segmentation for 3D Gaussian Splatting

Sungjae Choi, Seunghee Koh, Junmo Kim

arXiv:2609.28645v1cs.CV

TL;DR

Existing multi-scale 3DGS segmentation separates geometry reconstruction from feature learning, leaving geometry unaware of semantic structure and relying on incomplete masks. PePESeg3D integrates perception priors into geometry reconstruction and contrastive learning. It reports state-of-the-art segmentation performance across multi-object, text-guided, and fine-grained tasks, as well as segmentation and reconstruction results on several benchmarks.

  • Problem

    Multi-scale 3DGS methods optimize geometry and segmentation features separately, leaving geometry semantically misaligned and mask supervision inconsistent or incomplete.

    Source: §1 · p. 2

  • Method

    PePESeg3D uses monocular depth and segmentation masks in geometry reconstruction, then applies dense depth-color cues and view-consistent centroid supervision in contrastive feature learning.

    Source: §1 · p. 2

  • Results

    PePESeg3D reports state-of-the-art performance across multi-object, text-guided, and fine-grained segmentation tasks, and state-of-the-art segmentation and reconstruction on SPIn-NeRF and NVOS.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    The reported results highlight the importance of perception-aware geometry optimization and feature learning for accurate, spatially coherent 3D segmentation.

    Source: §1 · p. 2, §6 · p. 14

    Cross-scale feature sharing can make nearby query scales similar, limiting separation of entities with almost identical physical scales.

    Source: I Limitation · p. 26

Abstract

from arXiv · show

Recent advancements in 3D Gaussian Splatting (3DGS) have extended its capabilities to multi-scale segmentation. Existing methods reconstruct a scene with Gaussian primitives and learn multi-scale segmentation features separately, which leaves the geometry unaware of semantic structure and the feature learning dependent on incomplete mask supervision. To address these limitations, we present PePESeg3D, a novel framework that injects perception priors into a multi-scale 3D Gaussian segmentation pipeline. To fully exploit perception priors, we integrate them not only into contrastive feature learning but also into the upstream geometry reconstruction. Specifically, PePE Reconstruction incorporates monocular depth and mask constraints to ensure semantically coherent object structures. Building on this aligned geometry, PePE Contrastive Learning leverages dense depth-color cues and view-consistent centroid supervision to compensate for the incompleteness of multi-scale masks obtained from a 2D foundation model. Extensive experiments on the SPIn-NeRF, LERF-Mask, and NVOS benchmarks demonstrate that PePESeg3D achieves state-of-the-art performance in both multi-scale segmentation and scene reconstruction, highlighting the importance of integrating perception priors into both geometry optimization and feature learning for accurate multi-scale 3D segmentation. Our code is available at https://github.com/BeCow5X5/PePESeg3D.

1 Introduction

PePESeg3D addresses multi-scale 3DGS segmentation’s semantic geometry misalignment and incomplete mask supervision by injecting perception priors into reconstruction and feature learning. The authors report state-of-the-art segmentation performance across multiple tasks and benchmarks.

Motivation

  • Sequential geometry and feature optimization leaves photometrically initialized Gaussians semantically misaligned, while SAM masks can be inconsistent across views and omit entities.These sparse, unreliable mask signals become more problematic across multiple granularities.Source: §1 · p. 2

Approach

  • PePE Reconstruction incorporates 2D segmentation masks and monocular depth into geometry optimization, aligning Gaussian primitives with semantic boundaries and 3D structure.The resulting geometry is intended to facilitate subsequent multi-scale feature learning.Source: §1 · p. 2
  • PePE Contrastive Learning uses dense depth-color cues to supervise feature distinctiveness where SAM masks are ambiguous or missing.View-consistent centroid supervision guides local features and encourages label coherence across viewpoints.Source: §1 · p. 2

Results

  • PePESeg3D achieves state-of-the-art performance across multi-object, text-guided, and fine-grained segmentation tasks, including SPIn-NeRF and LERF-Mask benchmarks.The authors also report state-of-the-art results in segmentation and reconstruction on SPIn-NeRF and NVOS.Source: §1 · pp. 2–3

2 Related Works

Radiance-field segmentation methods use either single-scale features or multi-scale representations, with the latter modeling granularity through discrete levels or continuous physical scale.

2.1 Single-Scale Segmentation in Radiance Fields

  • Feature-3DGS distills SAM encoder features into Gaussians, Gaussian Grouping uses tracking-generated dense segmentation maps, and Unified-Lift learns instance identities end to end.These are examples of single-scale segmentation approaches in radiance fields.Source: §2.1 · p. 3

2.2 Multi-Scale Segmentation in Radiance Fields

  • Multi-scale methods use discrete SAM granularity levels or continuous physical scale; PePESeg3D builds on continuous scale to enforce structural consistency across views and granularities.Examples include OmniSeg3D’s hierarchy of up to four levels, Click-Gaussian’s fine/coarse features, GARField’s continuous conditioning, and SAGA’s scale-gated features.Source: §2.2 · p. 3

3 Preliminaries

The preliminaries describe 3DGS rendering, a mask-based Gaussian refinement procedure, and scale-aware features conditioned on physical scale.

3.1 3D Gaussian Splatting

  • 3DGS represents scenes with trainable Gaussian primitives and renders pixel color by alpha blending depth-sorted Gaussians projected onto the image plane.The Gaussian parameters are optimized with photometric loss between the input and rendered image.Source: §3.1 · pp. 3–4

3.2 Gaussian Refinement

  • Gaussian refinement moves a primitive overlapping a binary-mask boundary fully inside the mask, approximating its 3D inner-mask ratio from the 2D projection.Under a local affine projection approximation, refinement changes the Gaussian’s mean and scale while keeping other attributes unchanged.Source: §3.2 · p. 4

3.3 Scale-Aware Feature

  • Scale-aware features modulate a Gaussian’s feature vector with a shared gate conditioned on normalized physical scale.The raw scale is normalized to [0,1] using the scene’s minimum and maximum scales.Source: §3.3 · p. 4

4 Methods

PePESeg3D uses SAM masks and monocular depth to guide Gaussian reconstruction, then optimizes scale-dependent features with mask, perceptual, and centroid supervision. Its method combines boundary-guided refinement, depth alignment, and several complementary contrastive objectives.

4.1.1 Gaussian Refinement Guided Initialization

  • Gaussians whose projected major axes intersect mask boundaries are refined in ascending mask-area order; refinement selects the top 10% with the smallest depth gaps.Refined Gaussians are excluded from later updates, and candidates are limited to those close to surfaces.Source: §4.1.1 · pp. 5–6

4.1.2 Monocular-Depth Constrained Learning

  • Monocular-depth supervision fits scale and shift to align rendered and relative depth, combines this loss with photometric loss, and pretrains geometry before feature learning.At iteration K, the aligned primitives undergo refinement-guided initialization.Source: §4.1.2 · p. 6

4.2 PePE Contrastive Learning

  • PePE Contrastive Learning optimizes Gaussian-attached features and a scale gate, using rendered-feature cosine similarity to pull similar pixel pairs together and push dissimilar pairs apart.Features are rendered by alpha blending for each query scale.Source: §4.2 · p. 6

4.2.1 Scale-Aware Contrastive Loss

  • For each pixel and query scale, scale-aware labeling selects the finest-grained covering mask whose physical scale meets the query, then groups pixels by shared labels or finer masks.This preserves small-object pixel similarity even when the query scale represents a larger entity.Source: §4.2.1 · pp. 6–7

4.2.2 Perception Contrastive Loss

  • Perception contrastive learning uses scale-expanding spatial windows and local depth and color statistics to pull pairs below both adaptive thresholds together and push pairs exceeding both apart.Pairs with both pixels inside valid mask regions are excluded, supplementing rather than interfering with mask supervision.Source: §4.2.2 · p. 7

4.2.3 View-Consistent Centroid Loss

  • View-consistent centroid loss uses uppermost- and lowermost-scale cluster means refreshed from random views, attracting only pixels sufficiently similar to their nearest centroid.Candidate means are merged when cosine similarity exceeds 0.9; pixels are retained when similarity to the nearest centroid exceeds 0.9.Source: §4.2.3 · p. 8

4.2.4 Final Objective

  • The final objective combines the proposed losses with weights and regularizes both Gaussian and aggregated image-feature norms toward unit length.The norm regularization is intended to prevent unconstrained large-norm features from dominating rendering.Source: §4.2.4 · p. 8

5 Experiments

Experiments evaluate PePESeg3D across datasets with coarse, fine-grained, and mixed-granularity annotations, using segmentation and reconstruction metrics. The method leads multi-scale segmentation comparisons across the reported benchmarks, improves reconstruction quality, and benefits from both reconstruction and contrastive-learning components.

5.1.1 Implementation Details

  • Reconstruction trains for 30,000 iterations, then freezes geometry for 10,000 contrastive-learning iterations; centroid sets refresh every 200 steps.Gaussian refinement starts at iteration 4,000; centroid-loss activation begins at iteration 7,000.Source: §5.1.1 · pp. 8–9

5.1.2 Datasets

  • Datasets span mixed-granularity SPIn-NeRF, intricate-detail NVOS, coarse LERF-Mask, and fine-part LERF-Mask-Fine, with dataset-specific splits for novel-view and unseen-view evaluation.SPIn-NeRF holds out every eighth image; LERF datasets use official splits with 150–250 training views and 3–4 test views.Source: §5.1.2 · p. 9

5.1.3 Evaluation Protocols

  • Evaluation uses label propagation for SPIn-NeRF and LERF-Mask, promptable segmentation for NVOS and LERF-Mask-Fine, and PSNR, SSIM, and LPIPS for reconstruction.Corresponding-view feature-similarity thresholds are 0.7 for label propagation and 0.9 for promptable segmentation.Source: §5.1.3 · pp. 10–11

5.2 Segmentation Results

  • On SPIn-NeRF, PePESeg3D leads with 92.3% mIoU and 98.9% mAcc; on LERF-Mask it exceeds SAGA by 2.1%p mIoU and 2.5%p mBIoU.Table 1 covers joint segmentation and reconstruction; Table 2 compares single-scale and multi-scale segmentation methods.Source: Table 1 · p. 10, Table 2 · p. 10, §5.2 · p. 11
  • On LERF-Mask-Fine, PePESeg3D reaches 70.8% mIoU, 1.2%p above SAGA; on NVOS, it leads with 92.2% mIoU and 98.5% mAcc.Single-scale methods score 15–23% mIoU on LERF-Mask-Fine; Table 4 compares NVOS segmentation methods.Source: Table 3 · p. 10, §5.2 · p. 12, Table 4 · p. 12

5.3 Reconstruction Results

  • PePESeg3D improves reconstruction across datasets, including a 1.22 dB gain over SAGA on NVOS, while suppressing artifacts and recovering background details.Figure 5(a) attributes removal of noisy Gaussians between the Trex skeleton and background wall to monocular depth priors.Source: §5.3 · p. 12, Table 5 · p. 12, Figure 5 · p. 13

5.4 Analysis

  • Ablations report reconstruction gains of 0.46 dB from Gaussian refinement and 0.76 dB from depth supervision, alongside segmentation gains from reconstruction, perception, and view-consistency losses.The reported mIoU gains are 0.8%p, 1.0%p, and 0.3%p, respectively; qualitative analyses show scale-adaptive clusters and consistent labels across views.Source: §5.4 · pp. 13–14, Table 6 · p. 13, Table 7 · p. 13

6 Conclusion

PePESeg3D integrates perception priors into geometry reconstruction and contrastive feature learning for multi-scale 3DGS segmentation. The paper reports strong performance across benchmarks, with state-of-the-art segmentation and reconstruction on SPIn-NeRF and NVOS.

6 Conclusion

  • PePESeg3D combines depth- and mask-guided reconstruction with depth-color cues and centroid supervision, and reports strong results across coarse and fine granularities.The conclusion reports state-of-the-art segmentation and reconstruction on SPIn-NeRF and NVOS, and superiority to prior multi-scale methods on LERF-Mask and LERF-Mask-Fine.Source: §6 · p. 14

A Additional Implementation Details for PePE Reconstruction

The implementation adapts depth supervision to scene geometry and refines Gaussians near semantic boundaries. It also improves multi-scale feature learning through targeted sampling, mask reweighting, and depth-based boundary handling.

A Additional Implementation Details for PePE Reconstruction

  • Depth supervision uses lower weight for 360° scenes, stronger weight for forward-facing scenes, and decays toward 0.02 during training.The decay relaxes geometric constraints later so optimization can focus on photometric error and high-frequency textures.Source: A Additional Implementation Details… · p. 20
  • Local alignment supplements global depth alignment by fitting scale and shift within each pixel’s window to capture local geometric inconsistencies.The enhanced depth loss combines global and local terms.Source: A Additional Implementation Details… · p. 20

B.1 Sampling Strategy

  • Hard-example mining oversamples pixel pairs grouped at a scale but not co-located in any finer mask, while uncovered pixels are excluded and Gaussian features are averaged with 16 nearest neighbors.These choices target scale-dependent grouping and local spatial consistency.Source: §B.1 · pp. 20–21

B.2 Reweighting Strategy

  • Pixel weights invert the average area of masks covering each pixel, and pair weights multiply the two pixel weights before min-max normalization to [1,10].The reweighting counters the dominance of frequently sampled large masks and keeps small-mask pairs represented.Source: §B.2 · p. 21

B.3 Implementation Details in Perception Loss

  • Depth gradients define dilated boundaries and their complement defines interiors; depth is preferred to RGB because RGB gradients can mark texture and lighting changes as false boundaries.The perception loss is restricted to valid interior regions to reduce noisy boundary supervision.Source: §B.3 · pp. 21–22, Figure 6 · p. 21

Detailed algorithm of PePE Reconstruction

  • PePE Reconstruction applies Gaussian refinement after gradient computation and density control, avoiding incorrect gradient propagation through newly refined Gaussians.The algorithm also supervises alpha-rendered depth to align Gaussians with scene geometry.Source: Detailed algorithm of PePE… · p. 23

D Per-Scale Analysis

The fixed-query-scale analysis shows fine-part accuracy peaking at the smallest tested scale, while the extracted clusters become fewer and larger as scale increases.

D Per-Scale Analysis

  • As query scale increases, HDBSCAN clusters fall from 83 to 43 while average mask size grows from 2,336 to 5,058 pixels.The paper describes this as fine parts progressively merging into their containing objects.Source: D Per-Scale Analysis · p. 23, Table 8 · p. 23
  • Fine-part accuracy peaks at s = 0.01 and decreases monotonically as query scale grows.Table 8 reports fine-part mIoU in its top portion.Source: Table 8 · p. 23, D Per-Scale Analysis · p. 23

E Efficiency Analysis

On SPIn-NeRF, PePESeg3D adds modest training and memory cost over SAGA while retaining comparable inference speed. The reported cost breakdown attributes much of the added segmentation time to perception loss.

E Efficiency Analysis

  • Measurements use a single NVIDIA RTX 4090 on SPIn-NeRF, and single-scale methods report joint reconstruction-and-segmentation training time.Multi-scale methods report the two sequential stages separately.Source: E Efficiency Analysis · p. 23, Table 9 · p. 24
  • PePESeg3D uses only 1 GiB more VRAM than SAGA, with modest additional training and comparable inference speed.Table 9 compares cross-method costs on SPIn-NeRF.Source: Table 9 · p. 24, E Efficiency Analysis · p. 24
  • Reconstruction adds under four minutes over SAGA, while segmentation adds 5.5 minutes.The comparison separates reconstruction and segmentation training for multi-scale methods.Source: Table 9 · p. 24, E Efficiency Analysis · p. 24
  • Compared with SAGA, PePESeg3D adds under four minutes of reconstruction, of which refinement and depth loss account for 1.54 minutes; segmentation adds 5.5 minutes, mostly from perception loss.Source: E Efficiency Analysis · p. 24

F Robustness to 2D Priors and Limited Views

PePESeg3D remains robust under weaker mask supervision, reduced view coverage, and changes to depth backbones. The reported hyperparameter variants also stay close to the default configuration.

F Robustness to 2D Priors and Limited Views

  • With SAM2 supplying 25–30% fewer masks and with only 1/4 of input views, PePESeg3D degrades less than SAGA; depth-backbone changes shift results by at most 0.8 mIoU and 0.9 mBIoU.Table 11 reports mIoU / mBIoU (%) on LERF-Mask-Fine; Depth-Anything-V2 performs best in the reported setting.Source: F Robustness to 2D Priors and Limited… · p. 24, Table 11 · p. 25

G Hyperparameter Sensitivity

  • On LERF-Mask, all tested variants stay within 0.5%p of the default, and changing centroid threshold or refresh period shifts results by under 0.5%p.Table 12’s default is k = 10%, τ = 0.90, T = 200, and Depth-Anything-V2; k = 5 leaves boundary Gaussians unrefined, while k = 20 admits off-surface Gaussians that perturb geometry.Source: G Hyperparameter Sensitivity · p. 24, Table 12 · p. 25

H More Qualitative Results

The appendix expands qualitative evaluation with uncropped reconstruction context and multi-view segmentation and decomposition visualizations. These examples emphasize view consistency and scale-dependent granularity.

H More Qualitative Results

  • Figures 8 and 10 present uncropped, high-resolution versions of visual results discussed in the main paper and Figure 7, providing full context and improving clarity.Source: H More Qualitative Results · p. 25
  • Figure 9 is identified as showing NVOS failure cases.The supplied caption does not specify the failures depicted.Source: Figure 9 · p. 26
  • Figure 11 shows PePESeg3D producing more view-consistent and accurate LERF-Mask and LERF-Mask-Fine segmentations than the illustrated baselines.OmniSeg3D-GS struggles with view consistency and can assign identical labels to semantically distinct objects, while SAGA may overlook fine details such as a small tag.Source: H More Qualitative Results · p. 26
  • Figure 12 compares SPIn-NeRF ground-truth masks with predictions and shows predictions from alternate viewpoints.The alternate views illustrate the feature field's robustness and consistency across perspectives.Source: H More Qualitative Results · p. 26
  • Figure 13 presents automatic scene decomposition at three scale levels, with HDBSCAN clusters shown in random colors.The visualized results demonstrate scale-dependent granularity and view consistency across perspectives.Source: H More Qualitative Results · p. 26

I Limitation

The limitations include cross-scale feature interactions, residual segmentation errors on highly complicated objects, and scene-dependent interpretation of normalized query scale.

I Limitation

  • Cross-scale gating can make nearby query scales' features similar, leaving entities with almost identical physical scales difficult to separate cleanly.The paper identifies isolating or controlling these interactions as a direction for future work.Source: I Limitation · p. 26
  • Objects with highly complicated structures may remain imprecisely segmented when SAM struggles to recognize them as a single entity.Perceptual priors help but cannot fully compensate for the backbone's zero-shot segmentation limitation.Source: I Limitation · pp. 26–27
  • Because physical scale is normalized within each scene, the same query scale can correspond to different absolute granularities across scenes.Choosing a query scale for a new scene requires a reference object or a short sweep; scene-agnostic scale parameterization is left for future work.Source: I Limitation · p. 27
Loading 2609.28645v1…