Source-linked AI summary
VCAR: Training-Free 3DGS Segmentation via View Completeness and Axis-Aware Boundary Refinement
Kun Cao, Di Wang, Haibin Zhu, Haozhi Huang, Xu Wang, Zheng Shi, Guanghua Yang
TL;DR
3DGS segmentation methods often require costly per-scene training and can produce blurred boundaries from incomplete viewpoints and anisotropic Gaussian overflow. VCAR replaces this with training-free coarse-to-fine voting, supplementary spherical views, and axis-aware refinement, achieving state-of-the-art results on NVOS and LERF. Its scope is limited by assumptions about object shape and conservative compression that can erode fine structures.
Problem
Existing 3DGS segmentation methods incur substantial per-scene training overhead, while limited viewpoint coverage and anisotropic Gaussian overflow contribute to blurred boundaries.
Method
VCAR uses training-free coarse-to-fine visibility-weighted voting, Spherical Spiral Sampling for supplementary object-centric viewpoints, and Axis-aware Boundary Refinement for targeted axis compression.
Results
VCAR achieves state-of-the-art accuracy and efficiency on NVOS and LERF, including 93.5% mIoU and 98.6% mAcc on NVOS and gains across all four LERF scenes.
Takeaways & Limitations
The framework provides a unified inference-based approach that improves viewpoint coverage and refines anisotropic boundary artifacts without per-scene training.
Takeaways & Limitations
The object-centric sphere assumes roughly convex targets, while conservative minimum-scaling in ABR can cause minor boundary erosion in fine structures and textures.
Abstract
from arXiv · showhide
Semantic segmentation in 3D Gaussian Splatting (3DGS) is crucial for advancing 3D scene understanding. Existing methods predominantly rely on feature distillation, which incurs substantial per-scene training overhead and often yields blurred segmentation boundaries. We identify that these boundary artifacts are driven in part by insufficient viewpoint coverage and boundary overflow of anisotropic Gaussian primitives. To address these challenges, we propose VCAR, a training-free coarse-to-fine segmentation strategy based on View Completeness and Axis-aware Boundary Refinement. In the coarse stage, a visibility-based weighted multi-view voting scheme rapidly localizes the target. In the fine stage, an object-centric sphere derived from the coarse result generates supplementary viewpoints via Spherical Spiral Sampling (SSS), allowing multi-view voting on the augmented views to precisely refine object boundaries and suppress irrelevant 3D Gaussians. Moreover, we introduce Axis-aware Boundary Refinement (ABR) to mitigate artifacts from anisotropic primitives. By decomposing the projected 2D covariance into per-axis contributions, ABR identifies the dominant axis responsible for boundary leakage and applies targeted anisotropic compression exclusively along that axis. Extensive experiments on NVOS and LERF demonstrate that VCAR achieves state-of-the-art segmentation accuracy and efficiency without training. Our code is available at https://github.com/DDKK0526/VCAR.
1 Introduction
VCAR addresses training overhead and blurred 3DGS segmentation boundaries with a training-free coarse-to-fine framework targeting viewpoint incompleteness and anisotropic boundary overflow.
- Existing 3DGS segmentation methods distill 2D semantic features into Gaussians through per-scene optimization, creating substantial computational overhead.These methods use foundation models such as SAM, CLIP, and DINO before additional 3D training.
- Limited and biased training viewpoints leave parts of target objects insufficiently constrained, producing boundary ambiguity.Anisotropic Gaussians can also extend beyond object surfaces in projection, causing aliased edges and floating fragments.
- VCAR uses visibility-based weighted voting on original training views to rapidly localize a user-specified target without training.The coarse stage provides the spatial prior for subsequent refinement.
- Supplementary viewpoints generated by Spherical Spiral Sampling improve view completeness for refined multi-view voting.The fine stage renders only coarsely segmented Gaussians, reducing inter-object occlusion before voting on augmented views.
- Axis-aware Boundary Refinement identifies the scale axis responsible for boundary overflow and compresses only that axis, preserving geometry in other directions.This targets anisotropic boundary artifacts rather than shrinking all Gaussian axes indiscriminately.
- VCAR’s reported contributions include state-of-the-art accuracy and efficiency on the NVOS and LERF datasets.
2 Related Work
Related 3DGS segmentation methods transfer 2D semantics through distillation or mask lifting, while VCAR unifies training-free view augmentation and axis-selective boundary refinement.
- 3D Gaussian Splatting: 3DGS represents scenes with explicit anisotropic Gaussian primitives and supports real-time rendering through projected covariance and front-to-back alpha blending.
- 3D Segmentation in 3DGS: 2D foundation models such as SAM, CLIP, and DINO provide semantic capabilities that recent methods transfer into 3DGS representations.
- Existing Segmentation Methods: Feature-distillation methods optimize semantic features per scene, while mask-lifting methods directly transfer 2D masks into 3D and focus on multi-view consistency.Examples include LangSplat, Feature3DGS, GaussianGrouping, Gaga, OmniSeg3D, and SAGA.
- Training-Free Mask Lifting: Training-free mask-lifting methods reduce training overhead through center projection, linear programming, graph cuts, visibility-guided voting, or graph diffusion.
- Boundary Refinement: Boundary ambiguity remains difficult because volumetric Gaussians can span foreground and background regions, motivating splitting, decomposition, and virtual-view post-processing.
- VCAR: VCAR jointly addresses insufficient viewpoint coverage and boundary overflow in a unified training-free framework using object-centric multi-view augmentation.
- VCAR: ABR traces boundary overflow to specific 3D scale axes and applies axis-selective anisotropic compression instead of indiscriminate isotropic compression.This is intended to preserve structural fidelity in well-behaved directions while improving boundary conformity.
3 Method
VCAR performs training-free coarse-to-fine 3DGS segmentation by combining visibility-based voting, object-centric viewpoint augmentation, and axis-aware boundary refinement.
- Coarse Segmentation Stage: VCAR first renders training views, segments them with SAM 3, and aggregates the masks through visibility-based weighted voting to obtain Gcoarse.Visibility is assessed per Gaussian so voting uses only views where the primitive is visible.
- Object-Centric Sphere Estimation: An object-centric sphere is estimated from Gcoarse using Gaussian positions, robust 3σ outlier rejection, and a radius scaled from the reference-camera distance.The scaling factor η is intended to help contain the target for generated viewpoints.
- View Completeness Assessment: VCAR measures angular coverage with K = 2000 Fibonacci-lattice directions and triggers SSS when the maximum angular gap exceeds a threshold such as 90°.If coverage is sufficient, supplementary sampling is bypassed for efficiency.
- Spherical Spiral Sampling: SSS samples supplementary cameras along a continuous spherical spiral, orients them toward the sphere center, and uses the augmented views for temporally coherent mask refinement.Rendering with Gcoarse reduces inter-object occlusion before voting is reapplied.
4 Experiments
VCAR is evaluated on NVOS and LERF using mIoU and mAcc from held-out views, with experiments covering accuracy, efficiency, qualitative behavior, and module ablations. It achieves strong benchmark performance while operating without training.
- Benchmark Results: VCAR achieves 93.5% mIoU and 98.6% mAcc on NVOS, surpassing GaussianCut by 1.0% mIoU.The forward-facing capture setup limits viewpoint diversity, making view completeness enhancement particularly effective.
- Benchmark Results: VCAR achieves state-of-the-art performance across all four LERF scenes, with gains of +15.6% on kitchen and +7.5% on figurines.It also improves over the next best method by +6.1% on ramen and +6.3% on teatime.
- Efficiency: VCAR completes inference in ∼30 seconds on NVOS and ∼2 minutes on LERF, compared with minutes to hours for representative baselines.Feature-distillation methods require 20 minutes to 3 hours of per-scene training, while LUDVIG and SAGD require ∼12–16 and ∼2–3 minutes per object.
- Qualitative Comparison: Across diverse object scales, VCAR produces cleaner boundaries and fewer floating artifacts than baseline methods.In cluttered scenes, augmented multi-view voting suppresses adjacent background Gaussians, while ABR compresses only overflowing axes for anisotropic objects.
- Ablation Study: Figure 6 shows progressive refinement: SSS reduces under-segmented regions, and ABR removes residual overflow for tighter boundaries.The visual ablation isolates the complementary effects of supplementary viewpoints and boundary refinement.
5 Conclusion
VCAR is a training-free coarse-to-fine 3DGS segmentation framework targeting insufficient view coverage and anisotropic boundary overflow. Experiments report strong accuracy with zero training and approximately two minutes of inference per object, while identifying scope limits for non-convex objects and fine structures.
- Conclusion: VCAR combines object-centric spherical spiral sampling, visibility-based weighted voting, and Axis-Aware Boundary Refinement to address blurred 3DGS boundaries.The framework improves angular coverage, aggregates visible multi-view labels, and compresses the scale axis linked to projected overflow.
- Limitations: The object-centric sphere assumes roughly convex targets, so SSS may provide inadequate coverage for highly non-convex or thin elongated objects.The limitation is stated for examples such as cables and poles.
- Limitations: ABR’s conservative minimum-scaling strategy can cause minor boundary erosion, particularly on fine structures and textures.Future work is proposed around shape-aware sampling and soft, per-view compression weighting.
Supplementary Material
The supplementary material formalizes view-completeness assessment by filtering invalid cameras, probing the sphere with a Fibonacci lattice, and measuring the worst-case angular gap. It motivates the 90° trigger as a hemisphere-scale blind spot and reports efficient computation.
- Camera Validity: Invalid training cameras are filtered before view-coverage computation because they may not effectively observe the target object.A camera is retained when its visible foreground ratio reaches the default threshold ρ_min = 0.2.
- Coverage Computation: The maximum angular gap Δmax measures the worst-case distance from any spherical direction to its nearest valid camera.The method uses K = 2000 Fibonacci-lattice test directions on the unit sphere.
- Fibonacci Lattice: Fibonacci sampling distributes test directions nearly uniformly by using equal-area latitude slices and successive golden-angle rotations.This avoids polar clustering and constructs the lattice in O(K) time.
- Coverage Computation: The coverage computation uses a cosine-similarity matrix followed by angular-distance and maximum reductions to identify the most under-observed direction.With K = 2000, one matrix product and reductions take only 2–3 seconds in scenes with hundreds of cameras.
- Coverage Trigger: The Δth = 90° trigger corresponds to a hemisphere-scale blind spot with solid angle 2π, providing a geometric criterion for activating SSS.Fibonacci probing directly measures the view-completeness quantity rather than relying on pairwise camera-angle statistics.
B Axis-Aware Boundary Refinement: Detailed Derivation
ABR detects projected Gaussian overflow, attributes it to the dominant contributing 3D axis, and computes a directional compression factor from the observed mask boundary. It reconciles corrections across views and applies them only with sufficient cross-view support.
- Overflow Detection: ABR considers an observation eligible when the projected Gaussian center lies inside the image and on the foreground mask.Overflow is recorded when at least one projected ellipse endpoint lies outside the image or maps to background.
- Overflow Detection: The endpoint test is directional, so ABR does not penalize an elongated Gaussian when both endpoints remain within foreground support.Projected eigenvectors depend jointly on all three 3D axes rather than directly representing individual local axes.
- Dominant-Axis Attribution: ABR identifies the dominant axis by maximizing its squared projected directional contribution w_d, rather than selecting the longest 3D axis.The contribution depends on the axis scale and its projected alignment with the overflow direction.
- Directional Compression: The compression factor is chosen so the post-compression projected radius matches the directional mask-boundary distance ℓ_u.The factor is dimensionless, uses the positive root, and balances projected variance in pixel-squared units.
- Multi-View Reconciliation: For each Gaussian-axis pair, ABR takes the smallest supported factor across overflow observations and clips it to [f_min, 1].This implements the most restrictive supported correction while preventing enlargement or excessive shrinkage.
- Multi-View Reconciliation: ABR applies a reconciled update only when overflow has sufficient cross-view support, preventing isolated mask errors from changing 3D scales.Supplemental spherical views are excluded from the support counts, which use aligned retained training-view masks.
C.1 Per-Stage Runtime Analysis
VCAR remains training-free and practical at inference time, with runtime concentrated in SAM mask generation, rendering, and voting on larger LERF scenes.
- All timings are averaged over objects in each dataset and measured on a single NVIDIA A100 GPU.
- Coarse-stage rendering on LERF takes approximately 25 seconds, compared with about 5 seconds for fine-stage rendering because later stages reuse cached camera information.The coarse stage additionally loads camera poses and metadata before rendering.
- SAM segmentation dominates LERF runtime, taking about 25 seconds in the coarse stage and about 30 seconds in the fine stage.ABR takes approximately 3 seconds on NVOS and 10 seconds on LERF.
- ~30 s per object on NVOS and ~120 s on LERF characterize the full training-free VCAR pipeline.Conditional SSS triggering and cache reuse across stages help maintain practicality.
C.2 Backbone Portability with SAM 2
VCAR transfers beyond its default SAM 3 backbone: SAM 2 produces similar masks and remains competitive on LERF, although difficult cluttered scenes show larger degradation.
- 97.0% SSIM and 28.5 dB PSNR measure overall agreement between SAM 2 and SAM 3 masks from the same rendered frames.The lowest agreement occurs in the more cluttered kitchen scene.
- 72.7% average mIoU on LERF is achieved by VCAR with SAM 2 under identical pipeline settings.Even in kitchen, the most challenging case, VCAR+SAM 2 reaches 68.3% mIoU versus 59.1% for LangSplatV2.
- 2.4 points in average mIoU and 0.4 points in mAcc separate SAM 2 from SAM 3.The authors use SAM 3 in the main paper because it achieves the stronger average result and directly supports text prompts.
- An intermediate fine-stage VWV threshold performs best: the selected interval is τfine ∈ [0.5, 0.7].Low thresholds admit more background evidence, whereas high thresholds can reject target Gaussians visible or correctly segmented in only part of the view set.
C.4 Boundary-Aligned Evaluation on Selected LERF Objects
On the eight LERF objects that trigger SSS, adding SSS or ABR improves all reported boundary metrics over Coarse Only, while Full performs best on every metric.
- The boundary-aligned evaluation covers eight LERF objects satisfying Δmax > Δth and therefore triggering SSS.The reported averages characterize this fixed, method-defined subset rather than the complete LERF benchmark.
- Full achieves the best result on every reported boundary metric, while +SSS and +ABR each improve all three metrics over Coarse Only.SSS adds evidence from under-observed directions, whereas ABR reduces projected overflow.
- Additional LERF examples show cleaner boundaries and fewer floating fragments under clutter and inter-object occlusion.
- Additional NVOS examples show improved recovery of under-observed regions and reduced boundary leakage in forward-facing captures.
D.1 Robustness to Imperfect 2D Masks
VCAR can dilute sporadic mask errors through visibility-normalized multi-view voting, but persistent 2D errors remain an upstream limitation; SSS views may also contain rendering artifacts.
- VWV limits the influence of an isolated incorrect mask because votes are normalized over views in which each Gaussian is visible.Figure D.1(a) illustrates noisy or inconsistent masks coexisting with a stable final segmentation.
- Persistent semantic errors across many views become multi-view consensus and are inherited by VWV.SSS cannot correct a repeatedly wrong mask, and ABR changes selected Gaussian scales rather than semantic labels.
- Boundary-axis compression may suppress visually meaningful Gaussian contributions when an attributed axis remains visually inconspicuous after alpha compositing.The diagnostic sweep varies the compression factor from 1.0, leaving the axis unchanged, to 0.0, completely compressing it.
- SSS-generated views are intermediate observations rendered using Gcoarse, so missing background content and coarse geometry can create unnatural appearance or rendering artifacts.Their utility is judged by downstream segmentation improvement rather than image-level photorealism.
D.3 ABR versus Visual Quality
ABR can improve geometric boundary tightness while compromising visual fidelity when compressed Gaussians contribute meaningfully after alpha compositing. The framework also cannot recover target parts removed during coarse voting, limiting recall for thin, weakly visible, or occluded regions.
- Visual-quality tradeoff: A geometrically tighter boundary does not necessarily produce a visually better rendering because anisotropic Gaussian tails may cross foreground boundaries without being visually prominent.
- Visual-quality tradeoff: ABR compression can suppress visually meaningful Gaussian contributions when projected tails are visually inconspicuous after alpha compositing.This exposes a tradeoff between reducing geometric overflow and preserving visual fidelity.
- Coarse-stage coverage: If coarse voting omits a target part, SSS and ABR cannot recover it from scratch, reducing recall for thin, weakly visible, or heavily occluded parts.The missing hind leg of Teatime–Sheep exemplifies persistent upstream mask omission caused by occlusion.