GARField: Group Anything with Radiance Fields

• GARField reconstructs posed-image scenes with a scale-conditioned affinity field that decomposes them into hierarchical 3D groups.Source: §1 · p. 2
• Physical scale lets the same point belong to different groups at different granularities, reconciling otherwise conflicting 2D grouping signals.Source: §1 · p. 2, Figure 2 · p. 2
• SAM mask candidates are assigned physical scales and distilled with a mask-membership contrastive loss to resolve inconsistencies across views and candidates.Source: §1 · p. 2
• Contrastive and containment supervision encourages affinity fields with transitivity and containment across scales.Source: §1 · p. 2
• GARField recursively clusters the optimized field from coarse to fine, producing detailed, 3D-consistent groups that often improve on input 2D masks.Source: §1 · p. 2

• 2D hierarchical segmentation methods construct multi-level groups from contours, edges, spectral clustering, or learned segmentation hierarchies.Source: §2 · p. 2
• Panoptic methods constrain grouping with categories, while SAM exposes ambiguity through multiple prompted masks without recovering consistent hierarchical scene groups.Source: §2 · pp. 2–3
• Existing 3D part-decomposition methods focus on individual objects, whereas GARField distills 2D information for full-scene hierarchical grouping.Source: §2 · p. 3
• NeRF segmentation methods generally recover flat semantic or instance labels, while interactive and SAM-based approaches do not automatically provide hierarchical, view-consistent groups.Source: §2 · p. 3
• GARField uses scale-conditioned contrastive features to distill conflicting masks into 3D, distinguishing its goal from feature reconstruction methods such as LERF [13].Source: §2 · p. 3, Figure 3 · p. 3

3.1. 2D Mask Generation

• SAM automatically supplies overlapping candidate masks, whose 3D physical scales are estimated from deprojected NeRF depth in a shared world-space.Source: §3.1 · pp. 3–4

3.2. Scale-Conditioned Affinity Field

• Scale conditioning allows conflicting masks containing the same point to coexist at different affinity scales instead of competing during training.Source: §3.2 · p. 4
• Users can query the field at a click point and scale, or recursively cluster embeddings at descending scales to obtain coarse-to-fine scene decompositions.Source: Figure 5 · p. 4, Figure 6 · p. 5

3.2.1 Contrastive Supervision

• The contrastive objective pulls ray features from the same mask together and pushes features from different masks apart using a margin, with supervision restricted within each image.Source: §3.2.1 · p. 4

3.2.2 Densifying Scale Supervision

• Continuous scale augmentation fills unsupervised intervals, while a containment loss preserves smaller-scale groupings at larger scales.Source: §3.2.2 · pp. 4–5

3.2.3 Ray and Mask Sampling

• Training samples multiple images and many rays per image, using inverse-log-area mask sampling and coordinated scales to balance supervision and increase positive pairs.Source: §3.2.3 · p. 5

3.3. Implementation Details

• The implementation uses a separate hashgrid-and-MLP grouping head, deferred rendering, and staged training after initial NeRF optimization.Source: §3.3 · p. 5

4. Hierarchical Decomposition

• GARField recursively clusters 3D features at decreasing affinity scales to construct a coarse-to-fine scene hierarchy.HDBSCAN can produce either 2D masks from rendered features or 3D pointcloud clusters.Source: §4 · pp. 5, 7
• The resulting hierarchy extracts complete objects from global clusters and decomposes them into meaningful local subparts at smaller scales.Examples include scene-wide clusters and progressively finer object groupings visualized with Gaussian Splats.Source: Figure 7 · p. 6
• Compared with SAM, GARField incorporates masks from multiple viewpoints in 3D, addressing missing small masks when objects are distant or clustered.This multi-view fusion is illustrated as a contrast between SAM’s automatic masks and GARField’s scale-conditioned affinity field.Source: Figure 8 · p. 7
• Global clustering initializes top-level scene nodes, after which leaf nodes are repeatedly reclustered as scale decreases until scale 0.The implementation uses smax=1.0 and reduces scale by 0.05 per iteration.Source: §4 · p. 7

5. Experiments

• GARField is evaluated on varied indoor and outdoor Nerfstudio and LERF scenes with annotated hierarchical groupings and selected ground-truth masks.The evaluation targets object hierarchy and consistency across views.Source: §5 · p. 7

5.1. Qualitative Scene Decomposition

• GARField produces high-fidelity hierarchical 3D groupings across diverse objects, from keyboard keys and plant parts to nested toy and vehicle components.Varying scale separates objects or their subparts, while renderings represent complete 3D models rather than 2D view segmentations.Source: §5.1 · p. 7
• The tree decomposition progressively breaks a statue from a top-level cluster into subgroups such as hair, legs, and torso.Arrows in the visualization denote parent-child relationships in the hierarchy.Source: §5.1 · p. 7

5.2. Quantitative Hierarchy

• GARField produces more complete 3D masks than SAM across viewpoints, with higher mIOU against multi-view human annotations, especially for fine groups.Table 1 evaluates coarse, medium, and fine masks for projected points across three viewpoints.Source: §5.2 · p. 8, Table 1 · p. 8
• GARField’s multi-view fusion yields higher-fidelity groupings than single-view SAM, while ablations show that scale conditioning and dense hierarchy supervision are necessary.Table 2 reports maximum-overlap mIOU against human annotations, and Fig. 9 shows affinity degradation under naive supervision.Source: Figure 9 · p. 7, §5.2 · p. 8, Table 2 · p. 8

6. Limitations

• If the 2D mask generator omits a desired group, GARField cannot recover that group in 3D.Limited or uneven viewpoints can also create artificial boundaries when no view contains the object in full.Source: §6 · p. 8
• Physical scale does not resolve all ambiguity when multiple groupings have similar size, such as objects inside a container.The method may also place differently sized object parts at separate tree levels, and greedy clustering can create spurious small groups deeper in the tree.Source: §6 · p. 8

7. Conclusion

• GARField distills multi-level masks into a dense scale-conditioned affinity field for hierarchical 3D scene decomposition.Scale conditioning lets the field learn groups from conflicting 2D inputs at multiple levels.Source: §7 · p. 8
• The resulting multi-granularity groupings support applications such as robotics, dynamic scene reconstruction, and scene editing.The conclusion specifically identifies assets at a multitude of granularities and tasks requiring multi-level groupings.Source: §7 · p. 9

A. Additional Results

• The appendix also provides videos of hierarchical decomposition, global clustering, and interactive selection to illustrate GARField’s 3D grouping behavior.These visualizations use Gaussian Splatting for presentation.Source: A. Additional Results · p. 11

A.1. Gaussian Splat Visualizations

• GARField visualizations use separately trained NeRF and Gaussian Splatting models, assigning affinity features to Gaussian centers for clustering and scene manipulation.The Gaussian representation is used for visualization and operations such as clustering, selection, and filtering.Source: §A.1 · p. 11

A.2. 3D Hierarchical Decomposition

• Additional results exhaustively visualize entire subtrees for selected regions of interest, showing how nodes recursively subdivide across the hierarchy.The selected regions include desk, dozer, and bouquet scenes.Source: §A.2 · p. 11

A.2.1 Full Tree Visualizations

• Nodes at the same tree depth need not share the same scale because intermediate nodes may be pruned.This affects how tree layers should be interpreted in the visualizations.Source: §A.2.1 · p. 11
• Recursive trees decompose objects into coherent subparts and fine details, but limited view coverage can separate different sides of the same object.Spurious background grouping and face-side separation are reported as limitations of the exhaustive trees.Source: §A.2.1 · p. 11

A.2.2 Compressed Tree Visualizations

• Compressed-tree videos show that lower layers represent semantically fine groups while higher layers correspond to coarser granularities.The compressed views merge nodes within each layer and color clusters distinctly.Source: §A.2.2 · p. 11

A.3. Multi-Scale Clustering

• Video renderings of multi-scale clustering demonstrate view consistency for global scene clusters followed by sub-object clusters.The videos accompany the corresponding image visualizations.Source: §A.3 · p. 11

A.4. Global Clustering

• Global clustering across all scene Gaussians groups large background regions such as roads or bushes at suitable scales.The appendix evaluates global clusters from scales 0 to 1 in increments of 0.05.Source: §A.4 · p. 11

A.5. Interactive Selection

• User clicks can select 3D points and retrieve multiple groups across scales, including an excavator, crane, and scooper from one click.Groups with large overlap are merged after querying across scales.Source: §A.5 · pp. 11, 13

B.1. Hierarchical Decomposition

• The hierarchy is built greedily with HDBSCAN, recursively splitting non-noise clusters and reinserting noise clusters afterward.This can produce disappearing regions in lower tree levels.Source: §B.1 · p. 13
• Voxel downsampling reduces Gaussian resolution during tree construction, after which chains with one parent and one child are pruned.The queried affinity scale determines the downsampling resolution.Source: §B.1 · p. 13

B.2. Treatment of Clustering Noise

• HDBSCAN noise clusters arise from geometric misalignment, boundary features, or noise in the learned affinity field.The method assigns each noise Gaussian to the nearest physical cluster in Euclidean space, producing more cohesive results than feature-space soft clustering.Source: §B.2 · p. 13

B.3.1 Ground Truth Annotation

• Ground-truth masks are annotated on randomly selected novel views at semantic levels ranging from object parts to whole objects.The number of hierarchy levels varies by scene complexity, such as two levels for the bear’s arm and whole bear.Source: §B.3.1 · p. 13

B.3.2 Complete Visualizations

• The view-consistency visualizations compare annotated hierarchical masks with the closest masks produced by SAM and GARField across five scenes.Each scene includes clicked points, ground-truth masks, and zoomed result comparisons.Source: §B.3.2 · p. 13

B.4.1 Ground Truth Annotation

• The completeness experiment targets recovery of all reasonable masks in richly hierarchical objects, including parts, subparts, complete regions, and whole objects.Ramen annotations include nori, egg, egg yolk, noodles, soup, and the entire bowl.Source: §B.4.1 · p. 14

B.4.2 Complete Visualization

• Global clustering shifts from fine parts at small scales to larger semantic groups as scale increases across bouquet, desk, sandwich, table, teatime, succulent, and living-room scenes.Examples include petals or leaves at small scales and whole bouquets, tables, or tile groups at larger scales.Source: Figure 13 · p. 15, Figure 14 · p. 16, Figure 15 · p. 17
• View-consistency experiments construct fine, medium, and coarse hierarchies corresponding to parts, objects, and complete scene regions.Examples include flower petals to bouquets, keys to keyboards, egg yolks to soup areas, and gun parts to the whole gun.Source: Figure 20 · p. 22, Figure 21 · p. 23, Figure 22 · p. 24
• GARField outperforms SAM for finer masks, hierarchical GARField recovers more layered groupings, and dense supervision improves grouping stability and thoroughness.Examples include tiny keyboard keys, the full ramen mask, and small cookie-bag labels.Source: Figure 25 · p. 27