LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenes

• 3D scene representations preserve appearance and geometry but are not suited for semantic tasks, motivating the transfer of 2D foundation-model features into 3D.Source: §1 · p. 1
• Optimization-based feature transfer requires iterative scene-specific reprojection-loss minimization, creating a need for faster and simpler alternatives.Source: §1 · p. 1
• The proposed learning-free process uplifts 2D features and semantic masks into Gaussian Splatting scenes and can be adapted to different feature types.Source: §1 · p. 1, Figure 1 · p. 1
• Graph diffusion converts coarse inputs such as scribbles or feature–text alignment scores into 3D segmentation masks using DINOv2-informed structure, without requiring SAM.Source: §1 · p. 2
• The method matches state-of-the-art performance on segmentation and open-vocabulary localization while being significantly faster than prior optimization-based approaches.Source: §1 · p. 2

• NeRF- and Gaussian Splatting-based semantic methods distill or learn features from models such as DINO, CLIP, SAM, and diffusion models for 3D tasks.Source: §2 · p. 2
• Most Gaussian Splatting approaches minimize reprojection loss, which is costly for scenes containing over a million Gaussians and creates memory limitations.Source: §2 · p. 2
• Existing direct-uplifting methods are often segmentation-specific or rely on learned networks, whereas LUDVIG is applicable to any 2D representation and is parameter-free.Source: §2 · p. 2
• Prior segmentation systems commonly depend on SAM masks and incur overhead from dense mask generation and cross-view mask matching.Source: §2 · p. 2
• For open-vocabulary localization, prior methods typically require dimensionality reduction or efficient embeddings and can take one to two hours for feature-map generation and 3D optimization, while LUDVIG is an order of magnitude faster with graph diffusion.Source: §2 · pp. 2–3

3.1. Background on Gaussian Splatting

• Gaussian Splatting represents a scene with 3D Gaussian densities whose rendered pixel values depend on opacity, projected density, and depth ordering.Source: §3.1 · p. 3

3.2. Uplifting 2D feature maps into 3D

• Each 3D Gaussian feature is formed by a rendering-weighted average of 2D features contributed by all view–pixel pairs containing that Gaussian.Source: §3.2 · p. 3, Figure 2 · p. 4
• The aggregation is a normalized transposed rendering operation implemented through forward Gaussian Splatting calls with feature vectors inside the CUDA renderer, without explicitly constructing W or D.Source: §3.2 · p. 3
• Unlike reconstruction-based uplifting, the proposed scheme avoids costly optimization and can be interpreted as a single pre-conditioned gradient step; additional iterations did not improve feature quality.Source: §3.2 · pp. 3–4

3.3. Enriching features by graph diffusion

• Graph diffusion builds Gaussian edges from 3D nearest-neighbor geometry and DINOv2 feature similarity, optionally using unary regularization to prevent leakage beyond an object instance.Source: §3.3 · p. 4
• Diffusion repeatedly propagates initial 3D features through the graph, projecting them toward its dominant eigenspace and supporting generic features or task-specific segmentation masks.Source: §3.3 · p. 4

4.1. Multi-view segmentation

• Multi-view SAM masks are uplifted and aggregated across views, then rendered and thresholded to produce target-frame segmentations.Different prompt sets are generated by uplifting and reprojection of the reference mask, and predictions are averaged for cross-view consistency.Source: §4.1 · p. 5
• DINOv2 features are uplifted from sliding-window maps and compared with reference foreground features to obtain segmentation by thresholding projected similarities.The features are dimensionally reduced and re-weighted by PCA eigenvalues to favor components associated with foreground objects.Source: §4.1 · p. 5
• Graph diffusion propagates reference-mask information through similar 3D features and spatial connectivity, producing DINOv2 segmentations comparable to SAM.The final mask retains nodes reachable after diffusion and applies the feature-based regularization weights before projection and thresholding.Source: §4.1 · p. 5

4.2. Open-vocabulary object segmentation

• CLIP features are aggregated across overlapping multi-resolution patches, uplifted into 3D, and converted into text-query relevancy scores for localization or segmentation.Segmentation either thresholds rendered relevancies directly or uses high-scoring pixels as prompts for a SAM mask.Source: §4.2 · p. 5
• DINOv2 graph diffusion refines CLIP relevancies so scores cover the target object while decaying at its borders, improving open-vocabulary segmentation.For semantic segmentation, the pipeline instead assigns each Gaussian the predefined text label with highest CLIP similarity after replacing OpenGaussian’s learned 3D feature stage.Source: §4.2 · p. 5, §4.3 · p. 5

5.1. Qualitative results

• About 2 ms per view and feature dimension, direct Gaussian Splatting aggregation produces detailed 3D DINOv2 representations from coarse patch-level features.The first principal component mainly captures the foreground, while subsequent components refine foreground representations and detail the background.Source: §5.1 · p. 6, Figure 3 · p. 6
• Graph diffusion filters spatially disconnected distractors and constrains leakage, progressively spreading through object structure to refine segmentation.In the Fern scene, unary regularization prevents diffusion through the trunk; in Flower and Trex, near-full coverage is reached after five steps.Source: §5.1 · p. 6, Figure 4 · p. 6

5.2. Multi-view segmentation results

• Across NVOS and SPIn-NeRF, LUDVIG achieves IoU comparable to state-of-the-art methods without gradient-based optimization, while DINOv2 with graph diffusion performs on par with SAM-based approaches.DINOv2 captures more complete complex objects than SAM but can include background noise; poor Gaussian Splatting reconstruction in specular scenes partly explains lower results than OmniSeg.Source: §5.2 · p. 7, Table 1 · p. 8
• Ablations show that 3D uplifting improves over single-view segmentation, while graph diffusion further benefits challenging 360-degree scenes.Pure geometric reprojection works on forward-facing scenes but fails on 360-degree scenes, and single-view patch features produce coarser masks.Source: §5.2 · p. 7, Table 2 · p. 8

5.3. Open-vocabulary object localization

• On LERF localization and segmentation, LUDVIG surpasses prior approaches while achieving a 5 to 10× speedup in feature generation and uplifting.The method uses CLIP relevancy scores refined by graph diffusion, with separate 2D and 3D segmentation evaluation protocols.Source: §5.3 · p. 7, Table 3 · p. 8, Table 4 · p. 8

5.4. Open-vocabulary semantic segmentation

• On ScanNetv2, replacing OpenGaussian’s quantization-based feature training with direct uplifting yields significant accuracy gains while being considerably simpler and faster.The evaluation follows OpenGaussian’s protocol and uplifts SAM masks generated from textual queries.Source: §5.4 · p. 8, Table 6 · p. 8

• LUDVIG provides a simple aggregation mechanism for transferring 2D visual representations into 3D while bypassing traditional optimization-based approaches.The method operates on already trained Gaussian Splatting representations and is implemented within CUDA rendering.Source: §6 · p. 8
• Graph diffusion combines spatial structure with DINOv2 similarity to turn coarse scribbles or CLIP relevancy maps into accurate 3D segmentation masks.The approach remains computationally efficient and supports SAM-free segmentation.Source: §6 · p. 8
• 3D feature quality remains constrained by scene reconstruction failures in highly specular or motion-blurred settings.Learning 3D features jointly with Gaussian Splatting reconstruction may provide regularization and improve scene geometry.Source: §6 · p. 8

A. Using LUDVIG for downstream tasks

• The downstream evaluation uses multiple 2D frames, viewing directions, and a Gaussian Splatting scene representation.Multiview segmentation begins from either reference-frame scribbles or a complete foreground mask.Source: A. Using LUDVIG for downstream tasks · p. 12

A.1. Multiple-view segmentation with SAM

• SAM masks are generated per frame from reprojected reference-mask points, aggregated into 3D, and rendered back into target views for thresholded prediction.The procedure samples three prompts repeatedly and averages ten SAM predictions per view.Source: §A.1 · p. 12

A.2. Multiple-view segmentation with DINOv2

• DINOv2 patch representations are converted into fine-grained pixel-level feature maps through overlapping crops, dimensionality reduction, upsampling, and aggregation.The resulting features are uplifted into 3D and classified using either cosine similarity to foreground features or logistic regression.Source: §A.2 · p. 13
• The DINOv2 segmentation classifier depends on the reference input: logistic regression is used for complete masks, while cosine similarity is used for scribbles.Logistic regression is more effective when the reference pixels cover the full object; scribbles can miss uncovered object parts.Source: §A.2 · p. 13

A.3. Enhancing segmentation with DINOv2 using 3D graph diffusion

• Graph diffusion addresses DINOv2’s difficulty separating distinct objects with similar features by incorporating 3D spatial structure.The process initializes coarse Gaussian weights from reference scribbles or masks and constructs graph edges from DINOv2 similarity.Source: §A.3 · p. 13

A.4. Open-vocabulary object localization

• Open-vocabulary localization computes CLIP–text relevancy directly on 3D Gaussians, then projects relevancies into 2D for localization and segmentation.Localization selects the highest-relevancy pixel after mean-filter smoothing with kernel size K = 5.Source: §A.4 · p. 14, §A.4.1 · p. 14

A.4.2. Object segmentation

• CLIP-based segmentation is refined either with SAM prompts from high-relevancy pixels or with DINOv2-based graph diffusion.Diffusion starts from highly relevant nodes and propagates to neighboring nodes with similar DINOv2 features, improving object coverage without extending to similar objects.Source: §A.4.2 · pp. 14–15

B.1. Runtimes for LUDVIG

• Feature generation takes from a few seconds to an hour depending on the backbone, image count, and model calls; LUDVIG’s experiments generally require 1–5 minutes.LangSplat’s feature generation can take more than one hour because it queries SAM across image grids and resolutions.Source: §B.1 · p. 15
• Uplifting is linear in image count and feature dimensions; uplifting 100 images with 40-dimensional features takes 9 seconds on average.The reported rate is 2 ms per feature dimension for a 724 × 986 image.Source: §B.1 · p. 15
• Gradient-based optimization for 3D feature distillation typically uses approximately 3,000–30,000 gradient steps, while optimization becomes intractable for high-dimensional CLIP and DINO features.It can take as little as a few seconds for low-dimensional SAM masks, whereas FMGS’s total training time is 1.4 hours.Source: §B.1 · p. 15
• Scene-level graph setup takes about 30 seconds for 600,000 Gaussians, while prompt-specific diffusion costs 1–2 seconds for one foreground mask and averages 18 seconds for LERF queries.The scene operation includes Euclidean-neighbor queries and DINOv2 similarity construction.Source: §B.1 · pp. 15–16
• LERF inference averages 8 seconds per scene with SAM and 0.8 seconds without SAM, whereas LangSplat’s feature-map generation exceeds one hour.The SAM cost depends on test images and text queries.Source: §B.1 · p. 16

B.2. Runtime comparisons

• Compared with SAM-based automatic-mask methods, LUDVIG offers faster feature generation and uplifting at the cost of slightly higher inference time.Graph diffusion adds approximately 1 second per query, reported as 14 seconds for Ramen’s 14 queries.Source: §B.2 · p. 16
• Replacing OpenGaussian’s quantization-based training with LUDVIG aggregation produces significantly faster uplifting while achieving notable accuracy gains.The comparison keeps the same 2D feature-map generation and evaluation protocols.Source: §B.2 · p. 16

C.1. Per-scene multi-view segmentation results

• DINOv2 with graph diffusion achieves results comparable to state-of-the-art and on par with methods leveraging SAM masks, without gradient-based optimization.The evaluations cover average IoU on SPIn-NeRF and NVOS, with Tables C and D reporting the benchmark results.Source: §C.1 · p. 17, Table C · p. 17, Table D · p. 17
• Compared with SAM, DINOv2 captures more complete complex objects but can also include background noise.The T-Rex example shows DINOv2 recovering the tail and ribs while also including part of the stairs.Source: §C.1 · p. 17
• Purely geometrical mask uplifting works on forward-facing LLFF scenes but fails on 360-degree scenes, likely because limited views yield suboptimal 3D reconstruction.The forward-facing scenes span Orchids to Fortress, while failures occur on Fork, Pinecone, Truck, and Lego.Source: §C.1 · p. 18
• 3D uplifting considerably boosts segmentation over single-view approaches, whose lower-resolution semantic features produce coarser masks.The uplifted representation improves foreground isolation relative to purely single-view processing.Source: §C.1 · p. 18
• Adding 3D graph diffusion improves difficult 360-degree scenes where 2D segmentation from uplifted DINOv2 features typically fails because it lacks 3D spatial information.The affected scenes include Pinecone, Truck, and Lego.Source: §C.1 · p. 18

D.1. Segmentation tasks

• Graph diffusion spreads reference-scribble weights through neighboring Gaussians and uses feature regularization to refine object boundaries while preventing leakage.Diffusion reaches large neighboring Gaussians quickly, later covers smaller edge Gaussians, and can run for arbitrary steps once the object is covered.Source: §D.1 · p. 18

D.2. Visual comparisons of uplifted features

• LUDVIG produces more fine-grained background reconstruction and smoother DINOv2 features than N3F’s learned 3D DINO features.The comparison is reported across scenes, notably the trex and horns examples.Source: §D.2 · p. 18

D.3. 3D features for ScanNet semantic segmentation

• Uplifting yields coherent and meaningful 3D semantic features even with frozen positions and disabled densification during Gaussian Splatting reconstruction.This result is shown for semantic features on ScanNetv2 derived from object-level CLIP features assigned to SAM segments.Source: §D.3 · p. 18

D.4. Comparison to GaussianEditor’s uplifting

• Unlike GaussianEditor’s aggregation, LUDVIG’s convex combinations of pixel features avoid assigning incorrect semantics to large Gaussians in highly specular scenes.GaussianEditor’s normalization ignores rendering weights, favoring large opaque Gaussians; the discrepancy is especially visible in Room.Source: §D.4 · pp. 18–19

D.5.1. DINOv2-guided graph diffusion

• Raw CLIP relevancy scores can include similar objects or surroundings, while SAM prompts can undersegment or select the wrong object.Low-resolution CLIP features blur boundaries, and highest-relevancy point prompts may not span the object or may land on another object.Source: §D.5.1 · p. 19
• Graph diffusion can still produce inhomogeneous weights for multi-part objects or surrounding weights for nearby objects, complicating automatic thresholding.These failure modes are illustrated with sheep and knife examples.Source: §D.5.1 · p. 19
• Combining graph diffusion with SAM helps select broad point prompts without covering similar objects, yielding more accurate segmentation.The combination addresses limitations observed when either graph diffusion or 2D SAM segmentation is used alone.Source: §D.5.1 · p. 20

D.5.2. Comparisons for open-vocabulary localization

• LUDVIG and LangSplat correctly localize all four example objects in the LERF comparison, whereas LangSplat is more precise for queries such as chopsticks.The figure compares LERF, LangSplat, and LUDVIG using predicted points and annotated bounding boxes.Source: §D.5.2 · p. 20, Figure I · p. 25
• LangSplat’s greater precision comes with expensive SAM preprocessing: full-image mask construction takes about 23s per image and averages 80 minutes per LERF scene.The masks are generated by querying SAM over a grid of image points.Source: §D.5.2 · p. 20
• The visual comparison evaluates localization using red model predictions against black dashed annotation boxes.The comparison is adapted from LangSplat’s website.Source: Figure I · p. 25