Zero-Shot 3D Plant Organ Segmentation with SAM3 and Semantic NeRFs

1 Introduction

• 3D point-cloud segmentation is a prerequisite for accurate trait extraction, while supervised methods rely on labeled data whose 3D annotation is labor- and time-intensive.Multi-view imaging also allows 2D annotations to be projected into 3D.Source: §1 · p. 1
• The proposed pipeline combines text-prompted SAM3 with semantic NeRFs to perform open-vocabulary 3D plant organ segmentation without manual annotations.The authors state they are not aware of prior work performing this task completely without manual annotations.Source: §1 · p. 2
• The pipeline lifts SAM3 masks into 3D through a neural radiance field, requires no annotated training images, and is evaluated on a new dataset spanning ten species.The authors also identify SAM3’s bimodal confidence distribution as simplifying deployment and multi-view fusion as an effective consensus mechanism.Source: §1 · p. 2

2 Related Work

• Supervised point-cloud methods address plant-specific challenges but require dense point-wise annotations; PSegNet reports up to 89.90% mIoU, while Organ3DNet reports 93.57% IoU and 94.69% mWCov on five species.NeRFs and 3D Gaussian Splatting can lift 2D predictions into view-consistent 3D representations, but plant methods still require trained per-species detectors or single target classes.Source: §2 · pp. 2–3
• SAM3 accepts text prompts to generate concept-conditioned instance masks without an external detector, unlike Grounded-SAM’s detection-then-segmentation pipeline.The paper combines this capability with neural radiance fields to segment plants in 3D without domain-specific annotation or model training.Source: §2 · p. 3

3.1 Overview

• Multi-view RGB images are posed with structure-from-motion and segmented with text prompts; the images, masks, and poses train a semantic NeRF that exports a labeled 3D point cloud.The method hypothesizes that correct labels from most views outweigh occasional per-view errors without explicit consistency enforcement.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 Text-Prompted 2D Segmentation (SAM3)

• SAM3 processes images independently, discards masks below confidence 0.5, and merges instance masks into semantic classes with leaf-over-stem-over-pot priority.Unclaimed pixels receive the background label, which is excluded from the downstream semantic loss via class weighting.Source: §3.2 · pp. 3–4

3.3 Semantic NeRF Training

• The semantic NeRF uses a separate semantic MLP head whose logits are rendered along rays using density-derived weights, while semantic gradients are detached from the density backbone.Training combines RGB, semantic, interlevel, and distortion losses; class weights exclude background pixels and slightly up-weight stems.Source: §3.3 · pp. 4–5

3.4 Point Cloud Extraction and Post-Processing

• The trained NeRF renders depth and semantic logits for every second training view at half resolution, which are back-projected into 3D and filtered to retain reliable foreground points.Filtering uses accumulation, foreground class, depth-gradient, and bounding-box criteria; post-processing keeps the largest connected component and removes sparse outliers.Source: §3.4 · p. 5

4.1 Datasets

• The Begonia maculata dataset contains six synchronized calibrated cameras, 150 manually pixel-annotated images, and a 1.5-million-point manually segmented 3D reference cloud.The semantic classes are leaf, stem, and pot; pixel-perfect masks provide an oracle upper-bound reference.Source: §4.1 · pp. 5–6
• The diverse dataset contains ten species captured with seven calibrated cameras and 101 images per plant, and evaluation is performed exclusively in 3D.Because annotation is on the reconstructed cloud, geometry errors such as missing thin stems are not penalized; reported IoU measures semantic labeling on reconstructed points.Source: §4.1 · p. 6

4.2 Evaluation Protocol

• For 3D evaluation, each ground-truth point receives the label of its nearest predicted-cloud neighbor, while oracle masks separate NeRF reconstruction errors from 2D segmentation errors.The protocol notes that 2D and 3D mIoU use different domains, so their deltas are indicative trends rather than like-for-like comparisons; stem-boundary annotation variability may account for 1–3 pp IoU variation.Source: §4.2 · p. 6
• +23.7 pp mIoUfg separates SAM3 from Grounded-SAM on the 150-image Begonia 2D benchmark, with the largest gap on stems at +39.0 pp.Both methods use identical text prompts and default configurations.Source: Table 1 · p. 7

5.1 2D Semantic Segmentation on Begonia

• SAM3 achieves +23.7 pp higher foreground mIoU than Grounded-SAM in 2D (0.84 vs. 0.60), with the largest improvement for stems (+39 pp).SAM3's errors are mainly missed detections, whereas Grounded-SAM has cross-class confusion, including stem precision of 42%.Source: §5.1 · p. 7

5.2 3D Semantic Segmentation on Begonia

• Lifting SAM3 masks into the NeRF improves mIoU by +8.9 pp over per-frame predictions, with a +12.4 pp gain for stems.Multi-view observations act as implicit majority voting; thin geometry and occlusion particularly affect single-view stem predictions.Source: §5.2 · p. 8
• SAM3 exceeds Grounded-SAM by +13.4 pp mIoU in 3D and reaches 95.9% of the oracle ceiling on Begonia.The 3D comparison evaluates three 2D mask sources after lifting them into the semantic NeRF.Source: §5.2 · p. 8, Table 2 · p. 9
• The oracle reaches 96.6% mIoU, while 3.4 pp of the gap to perfection remain attributable to NeRF boundary smoothing.The passage attributes 4.0 pp of the total 7.4 pp gap to SAM3 mask imperfections, mostly for stems.Source: §5.2 · p. 9
• SAM3 results remain identical for thresholds τ ≤0.5, and higher thresholds reduce foreground coverage without improving prediction quality.Cross-class misclassification stays below 0.8% at all thresholds; the 3D curve degrades less than 2D due to multi-view redundancy.Source: Confidence Threshold Ablation · p. 10, Figure 4 · p. 10

5.3 Cross-Species Generalization

• Across ten species, mean mIoU is 0.856 ± 0.097, with leaf IoU above 0.91 and pot IoU above 0.90 for every species tested.The cross-species table reports ≥0.90 mIoU on four species without per-species tuning.Source: §5.3 · p. 10, Table 3 · p. 11
• Stem IoU has the highest cross-species variance (0.667±0.251), and dense, tangled branching is associated with lower performance.Single additional prompts also segment Sweet Potato dead leaves (IoU = 0.729) and Strawberry flowers (IoU = 0.669) without pipeline changes.Source: §5.3 · p. 11
• Pot points can be mislabeled as stem near soil-stem emergence, where limited camera coverage, elevated stem weighting, and missing substrate masks compound ambiguity.The resulting false positives form a thin planar layer on the pot surface that is described as trivial to remove in post-processing.Source: §5.3 · p. 11

6 Discussion

• +8.9 pp (SAM3) and +19.2 pp (Grounded-SAM) gains from 2D→3D support multi-view fusion as a self-correcting mechanism for lifting 2D labels into 3D.SAM3’s missed detections are often recovered through multiview redundancy, whereas Grounded-SAM cross-class confusions can yield incorrect 3D classifications.Source: §6 · pp. 11, 13
• Organ3DNet reports 93.57% IoU on a fully annotated 5-species, 889-sample dataset; the zero-shot pipeline achieves 92.6% mIoU on Begonia, placing it in the same performance range, though results are not directly comparable across datasets.Source: §6 · pp. 12–13
• Figure 5 contrasts Begonia predictions with ground truth and pairs RGB with predicted semantic point clouds for cross-species results under identical settings.The cross-species examples use no species-specific tuning.Source: Figure 5 · p. 12
• The oracle reaches 96.6% mIoU, leaving a 3.4 pp gap; the largest remaining gap is for stem at 8.6 pp.The discussion favors boundary-aware NeRF representations over better 2D masks for further improvement.Source: §6 · p. 13
• Across eleven morphologically diverse species, leaf IoU is ≥0.91 (mean 0.967) and pot IoU is ≥0.90 (mean 0.958) with identical prompts and no species-specific adaptations.Adding “dead leaf” or “flower” prompts detects those classes at IoU 0.729 and 0.669, respectively.Source: §6 · p. 13

• The pipeline produces semantic classes but lacks instance segmentation, and it can degrade or fail when SAM3 segmentation is insufficient for a species, growth stage, or prompted class.SAM3 is described as a single point of failure for these cases.Source: §7 · p. 13
• Hash-grid NeRF limitations complicate stem segmentation, while thin or occluded stems may be reconstructed thinner than the plant or with holes, affecting downstream trait extraction.The pipeline is limited to settings with structure-from-motion poses and multi-view 2D data.Source: §7 · pp. 13–14
• Training takes approximately 35 minutes per scene on an A5000 GPU, acceptable for single-scan phenotyping but unsuitable for high-throughput workflows needing rapid feedback.The authors identify 3D Gaussian Splatting as a direction to explore for speed and multi-plant scaling.Source: §7 · p. 14

• 0.856 ± 0.097 mIoU across ten diverse species; the controlled Begonia testbed reached 92.6% mIoU, or 95.9% of the oracle upper bound.Leaf and pot segmentation remained robust across species.Source: §8 · p. 14
• The pipeline uses multi-view images and class names without labeled training data, per-species fine-tuning, or domain-specific preprocessing; “dead leaf” and “flower” were added by one prompt without retraining.The reported multi-species evaluation used identical pipeline settings.Source: §8 · p. 14
• The oracle analysis attributes the remaining gap to SAM3 mask errors (4.0 pp, concentrated in stems) and NeRF boundary smoothing (3.4 pp), with boundary-aware NeRFs suggested as the higher-gain target.The authors also identify multi-view fusion as an effective consensus mechanism for imperfect zero-shot 2D predictions.Source: §8 · p. 14
• The results support annotation-free 3D plant organ segmentation as feasible and approaching supervised-method performance, with potential for flexible 3D phenomics pipelines across crops, experiments, and imaging systems.Source: §8 · p. 14