PaperlayerView exact revision on arXiv

Source-linked AI summary

ArborSplat: Online Semantic Gaussian Splatting SLAM for Orchards

Alessandro Masini, Matteo Frosi, Mirko Usuelli, Matteo Matteucci

arXiv:2609.26315v1cs.RO

TL;DR

Thin structures make image-to-3D semantic transfer unreliable, while appearance-driven mapping can underallocate capacity to important classes. ArborSplat jointly optimizes appearance and semantics online, fuses gated multi-view evidence, and reserves refinement capacity by class. On completed segments, it exceeds SGS-SLAM and GS3LAM in training-view and held-out mIoU while running faster.

  • Problem

    Image-to-3D semantic transfer is unreliable for thin structures, and reconstruction-driven densification can favor large or textured regions over underrepresented classes.

    Source: §I · p. 1

  • Method

    ArborSplat jointly optimizes appearance and semantics online on a Gaussian map, fuses multi-view evidence into a gated semantic point cloud, and reserves refinement capacity for important classes.

    Source: §I · p. 1

  • Results

    0.23 to 0.50 training-view and 0.15 to 0.36 held-out mIoU gains over SGS-SLAM and GS3LAM, with 1.7 to 7.5 times faster runtime on completed segments.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The semantic point cloud is more accurate than naive transfer on all six sequences, while reduced-budget refinement raises training-view fruit IoU in all six paired runs.

    Source: §VI · p. 8

Abstract

from arXiv · show

Orchard robots need maps that preserve small but semantically important structures such as trunks, trellises, and fruit. 3D Gaussian Splatting (3DGS) SLAM achieves high photometric fidelity. However, its optimization remains appearance-driven, and transferring image semantics to 3D points is unreliable for thin structures, whose pixels may receive depth from background surfaces. We present ArborSplat, an online semantic 3DGS SLAM system that tracks with LiDAR odometry and optimizes semantics directly on the Gaussian map, constrained by class-specific height bands above a ground plane fitted to each keyframe's stereo point cloud, and fuses multi-view evidence into a semantic point cloud online while rejecting labels inconsistent with the local ground surface or with monocular depth. Class-constrained refinement reserves Gaussian capacity for underrepresented structures and, under reduced budgets, increases training-view accuracy on tree classes. We evaluate the approach on apple and pear orchards during dormancy, flowering, and harvesting. On full routes, it keeps ATE below 0.5 m on all 12 traversals. On shared 301-frame segments, it exceeds SGS-SLAM and GS3LAM by 0.23 to 0.50 training-view and 0.15 to 0.36 held-out mIoU while running 1.7 to 7.5 times faster, whereas SemGauss-SLAM runs out of GPU memory on all six.

I. INTRODUCTION

Orchard robots need semantic, spatial maps that preserve small structures across seasons, but existing orchard Gaussian maps prioritize appearance and image-to-3D semantic transfer can fail on narrow structures. ArborSplat combines online semantic mapping with class-aware refinement and evaluates it across orchard species and growth stages.

I. INTRODUCTION

  • Orchard tasks depend on small structures such as woody growth, support wire, canopy, trellises, and fruit, motivating semantic and spatial maps that work in every season.The paper names pruning, spraying, and harvesting as examples.Source: §I · p. 1
  • Existing orchard Gaussian maps encode geometry and appearance but lack the explicit semantics needed for higher-level robotic reasoning.3DGS enables incremental, efficient photorealistic mapping within SLAM systems.Source: §I · p. 1
  • After multi-view transfer, mIoU falls from 66.1 to 48.1; narrow classes degrade, motivating fusion with ground-surface and monocular-depth label gates.Peters et al. report the transfer result; ArborSplat uses the stated gates to address this problem.Source: §I · p. 1
  • Finite-budget densification can favor large or textured regions, so ArborSplat reserves refinement capacity for underrepresented semantic classes.Conventional densification allocates primitives according to reconstruction gradients.Source: §I · p. 1
  • ArborSplat jointly optimizes appearance and semantics online in the Gaussian map without post-run refinement, and provides a semantic point cloud without 3D annotation.The point cloud fuses multi-view evidence and gates labels using the local ground surface and monocular depth; it is more accurate than naive transfer on all six sequences.Source: §I · p. 1
  • The evaluation compares class-resolved training-view and held-out-frame results across two species and three phenological stages against AgriGS-SLAM and three semantic Gaussian SLAM systems.The supplied evaluation scope names the systems and orchard conditions, but not their results.Source: §I · p. 1

II. RELATED WORK

Prior orchard Gaussian mapping captures geometry and appearance, while semantic Gaussian mapping and capacity allocation have been explored in other settings. ArborSplat's pipeline combines LiDAR-based tracking with online semantic mapping, addressing the risk that thin-structure pixels inherit background depth.

II. RELATED WORK

  • AgriGS-SLAM brought Gaussian mapping to orchards using LiDAR odometry and multi-camera reconstruction, but its map encodes geometry and appearance only.Earlier dense 3DGS SLAM work focused on indoor monocular or RGB-D scenes.Source: §II · p. 1
  • Prior semantic Gaussian SLAM systems optimize semantics online in indoor RGB-D scenes, while prior task-conditioned and horticultural capacity methods do not reserve capacity for several classes at once.Semantic features in Gaussian maps follow work on Semantic-NeRF and learned Gaussian semantic and instance features,.Source: §II · pp. 1–2

III. METHOD

  • The overview traces LiDAR tracking and keyframe selection through height-constrained image labels, Gaussian mapping, and a separately updated semantic point cloud.The point-cloud process lifts class probabilities to stereo points, filters trunk observations using monocular depth, votes in 10 cm voxels, and relabels points near ground.Source: Figure 1 · p. 2

II. RELATED WORK

  • Thin trunks and trellis posts can inherit depth from background surfaces, so visibility filtering may miss their wrong labels and multi-view agreement may reinforce them.This contrasts with the shared-surface assumption used when projecting image predictions onto 3D points.Source: §II · p. 2

III. METHOD

  • LiDAR odometry supplies camera poses and keyframes, and each Gaussian carries position, covariance, opacity, appearance, and semantic logits.A keyframe is optimized after its prediction is available, and past keyframes are not optimized again.Source: §III · p. 2

A. Geometry-Constrained Semantic Supervision

ArborSplat constrains prompt-based labels with orchard geometry, then renders and optimizes semantic states on Gaussians. Its mapping design normalizes low-opacity semantic logits and keeps semantic loss weight from being learned down.

A. Geometry-Constrained Semantic Supervision

  • A per-keyframe stereo point cloud supplies a RANSAC-fitted ground plane for class-specific pixel height bands, with implausible plane estimates refit or replaced by an earlier-keyframe median.Out-of-band classes at valid, in-range stereo pixels are vetoed, allowing the pixel to take its best remaining class.Source: A. Geometry-Constrained Semantic… · p. 2
  • Connected trellis components are retained only when their tallest valid pixel reaches a minimum height; components without valid in-range depth abstain.This addresses the difficulty of separating ground-starting trunks and posts with pixel height bands alone.Source: A. Geometry-Constrained Semantic… · pp. 2–3

B. Semantic Gaussian Mapping

  • Only Gaussians sharing a 0.1 m voxel with current-keyframe stereo points remain active on the GPU; the others stay dormant on the CPU.The semantic point-cloud fusion process does not modify the Gaussian semantic state.Source: B. Semantic Gaussian Mapping · p. 3
  • Each Gaussian carries view-independent semantic logits, rendered as a per-pixel class field in a second rasterization pass over the same geometry and opacity.The semantic state uses C class values per primitive and requires no decoder.Source: B. Semantic Gaussian Mapping · p. 3
  • Low accumulated opacity pushes composited logits toward zero and class probabilities toward uniform, especially on thin structures, so the logits are normalized before softmax.The normalization leaves fully opaque pixels unchanged.Source: B. Semantic Gaussian Mapping · p. 3
  • The photometric loss (0.8 L1 + 0.2(1 − SSIM)) and semantic loss use homoscedastic uncertainty weighting; semantic log-variance is fixed at −2.5, giving weight about 6.1, while the photometric one stays learnable.Source: B. Semantic Gaussian Mapping · p. 3

C. Semantics-Aware Map Refinement

ArborSplat allocates refinement capacity by semantic class rather than relying only on reconstruction-gradient rank. Floors help tree-class training-view accuracy under reduced budgets, but reduce it at the full budget.

C. Semantics-Aware Map Refinement

  • Candidates are assigned classes from the current keyframe's corrected label map, since alpha-composited Gaussian logits yield diffuse argmax labels.Only visible primitives accumulate densification gradients.Source: C. Semantics-Aware Map Refinement · p. 3
  • The allocator bounds each class by its demand and optional cap, reserves floor-share slots, then fills remaining capacity by global gradient rank within those bounds.Unused reservations return to the global pool; a cap can leave the refinement below budget when other classes lack demand.Source: C. Semantics-Aware Map Refinement · p. 3
  • Under reduced budgets, class floors increase training-view accuracy on tree classes, whereas at the full budget they reduce it; accordingly, the main configuration uses unconstrained allocation.Source: C. Semantics-Aware Map Refinement · p. 3

D. 3D Label Transfer

ArborSplat relabels the semantic point cloud after each keyframe to address unreliable first-view labels on thin structures. Its pipeline rejects depth-inconsistent trunk observations, fuses voxel evidence across views, and corrects near-ground labels.

D. 3D Label Transfer

  • After every keyframe, the helper rejects trunk observations behind monocular depth, votes across remaining observations per voxel, then relabels near-ground voxels as ground.The stages address errors from thin-structure pixels taking depth from the surface behind them and from views agreeing on incorrect labels.Source: D. 3D Label Transfer · p. 3
  • Trunk observations more than 0.5 m behind the estimated monocular surface are discarded after monocular depth is fitted to stereo depths.The threshold was selected most often by two-fold temporal cross-validation on reference frames in earlier experimental runs.Source: D. 3D Label Transfer · p. 3
  • Observations in each 10 cm voxel are fused by evidence-weighted voting, with weights increasing with segmentation confidence and decreasing with range beyond 3 m.The class share of accumulated evidence is adjusted to offset dominant classes; the passage cautions that β = 1 can overemphasize rare classes.Source: D. 3D Label Transfer · pp. 3–4
  • The final height gate relabels non-ground voxels below 0.15 m above local ground as ground, except in cells with fewer than ten ground-labeled voxels.Local ground is the median height of ground-labeled voxels in a 1 m2 horizontal cell; the threshold separates trunk labels projected onto ground behind the row from those on the row.Source: D. 3D Label Transfer · p. 4

A. Data and protocol

The protocol evaluates six orchard sequences across two species and three phenological stages, using inline-camera training and validation traversals with ground-truth poses. It reports photometric, semantic, trajectory, and repeatability metrics under specified evaluation procedures.

A. Data and protocol

  • The six-sequence dataset covers apple and pear orchards at dormancy, flowering, and harvesting, with training and validation traversals and ground-truth poses.Experiments use only the inline camera; adding the two side cameras did not improve semantics in any test.Source: A. Data and protocol · p. 4

B. Metrics

  • Photometric quality is measured by PSNR, SSIM, and LPIPS on training keyframes after optimization and validation keyframes rendered from the completed map.Validation uses 5 to 20 keyframes per sequence, and black sky in the target penalizes rendered content there.Source: B. Metrics · p. 4
  • Semantic mIoU pools per-class IoU across each sequence’s evaluated pixels, reporting all-six-class and structural mIoU while omitting classes absent from both reference and prediction.Pixels with opacity below 0.1 are assigned to sky; cloud scores project voxel centers with a z-buffer onto covered pixels.Source: B. Metrics · p. 4
  • ATE is the RMSE of estimated positions against ground truth after SE(3) Umeyama alignment without scale correction, with keyframe positions for ArborSplat and AgriGS-SLAM and every-frame inline-camera positions for semantic baselines.Training and validation traversals are aligned separately, and odometry stalls remain in the error.Source: B. Metrics · p. 4
  • Three pear-harvesting repeats on the same machine establish metric-specific noise floors; ablation changes beyond three standard deviations count as effects, and trajectories are deterministic across repeats.The reported noise floor ranges from 36% of the mean for held-out trunk IoU to 1.6% for ground.Source: B. Metrics · p. 4

C. Baselines and ablations

The study compares ArborSplat with photometric orchard SLAM and semantic Gaussian SLAM baselines, then specifies component ablations and class-budget tests. Implementation uses a shared GPU/CPU setup and class-specific geometric constraints.

C. Baselines and ablations

  • AgriGS-SLAM is rerun on all six sequences with keyframes every 1.0 m or 15° and a guarded logging fault, while semantic baselines process the same 301 frames per sequence.Semantic baselines receive inline images, stereo depth, and segmenter labels, but not ArborSplat’s height prior, fusion, or gates; their own poses are estimated.Source: C. Baselines and ablations · p. 4
  • Pear-harvesting ablations remove or replace individual components, while class-budgeted refinement tests ground caps and trunk/trellis floors at 75k and floor shares under 6k and 12k budgets.The reduced-budget floor-share tests use three paired runs each; the 75k variants use one run each.Source: C. Baselines and ablations · pp. 4–5
  • On shared 301-frame segments, the semantic Gaussian SLAM comparison covers training and held-out mIoU and structural mIoU, training ATE, and wall-clock time.Every system estimates its own poses; the caption identifies the comparison scope but provides no cell values here.Source: Table I · p. 5
  • All methods receive the same images, stereo depth, and segmenter labels; the table marks best results in bold and uses a dash for no completed run.The supplied passage does not report the table’s numerical outcomes.Source: C. Baselines and ablations · p. 5

D. Implementation details

  • Ground planes are fitted to stereo points with tilt and camera-height constraints, and class-specific height bands govern canopy, trunk, and trellis components.Canopy minima are 0.7 m for pear and 0.6 m for apple; trunk maxima are 1.0 m for pear and 0.9 m for apple, and trellis components must reach 0.8 m.Source: D. Implementation details · p. 5

V. RESULTS

Across orchard routes and shared segments, ArborSplat maintains low trajectory error and improves semantic accuracy over the completed semantic baselines. Full-route results also show a rendering–semantics trade-off and modest gains from online label fusion.

A. Comparison with Semantic Gaussian SLAM

  • +0.23 to +0.50 mIoU on training views and 0.15 to 0.36 mIoU on held-out frames, ArborSplat exceeds semantic baselines on completed segments.On training views, trunk IoU is 0.44 to 0.62 versus at most 0.12 for any baseline; ArborSplat is also 1.7 to 3.7 times faster than GS3LAM and 5.0 to 7.5 times faster than SGS-SLAM where they complete.Source: A. Comparison with Semantic Gaussian… · p. 5

B. Full-Route Mapping across Species and Seasons

  • Below 0.5 m ATE on all 12 full-route traversals, ArborSplat also beats AgriGS-SLAM on all six training traversals and four of six validation traversals.AgriGS-SLAM loses track on apple harvesting in replay; Table II reports full-route ATE and semantic results.Source: B. Full-Route Mapping across Species… · p. 5, Table II · p. 6

A. Comparison with Semantic Gaussian SLAM

B. Full-Route Mapping across Species and Seasons

  • Learning semantics costs 1.39 dB of training PSNR on average and 0.63 dB on validation, while reported PSNR exceeds the general-purpose systems’ values on 44 of 48 per-sequence comparisons.Those systems’ evaluation excludes sky error; Table III reports single-camera ATE and mean training- and novel-view PSNR per sequence.Source: B. Full-Route Mapping across Species… · p. 7, Table III · p. 7
  • Held-out mIoU is 0.317 to 0.356 versus 0.614 to 0.758 on training views across full routes, and one prompt set serves both species and all stages.Trellis posts remain intact across stages, while isolated fruit and thin branch tips lose the most render detail.Source: B. Full-Route Mapping across Species… · p. 7
  • The online semantic point cloud exceeds naive first-view transfer on every sequence, with structural mIoU gains of +0.007 to +0.013 on 4.5 to 5.8% of reference pixels.Table II covers structural mIoU of the semantic point cloud using naive transfer and the online fusion.Source: Table II · p. 6, B. Full-Route Mapping across Species… · p. 7
  • Table IV presents pear-harvesting ablations, with the first row giving the three-run ArborSplat mean and standard deviation and other rows giving one-run changes from that mean.Bold indicates changes beyond three standard deviations; the supplied passages do not provide the ablation values.Source: Table IV · p. 7

C. Ablation Study

The ablations show that class-budgeted refinement helps tree accuracy only under reduced budgets, while semantic optimization trades map size and optimization steps for semantic supervision. Height priors and a fixed semantic weight protect tree classes, and opacity normalization chiefly benefits held-out structural mIoU.

C. Ablation Study

  • Removing the height prior lowers held-out canopy IoU by 0.041 and training-view trunk IoU by 0.103; learning the semantic weight trades 0.94 dB PSNR for 0.104 training-view trunk IoU.The learned semantic weight decays to 0.28, about one sixth of the color weight.Source: C. Ablation Study · p. 7
  • Removing opacity normalization lowers held-out structural mIoU by 0.040, while training views barely change.Source: C. Ablation Study · p. 7
  • Under reduced budgets, floors raise training-view tree mIoU by a mean +0.05 at 6k and +0.03 at 12k, but lower it by 0.024 at 75k.Fruit IoU rises in all six reduced-budget pairs; mean held-out tree mIoU changes by +0.001 and −0.006, so the full-budget system uses unconstrained allocation.Source: C. Ablation Study · pp. 7–8
  • With semantics, the mapper completes 8 to 28% fewer steps in the same time and final maps are 1.16 to 1.57 times smaller, holding 68 to 95 million Gaussians.Wall-clock runtime differs from OURS-NS by −7.2% to +2.6% (mean −0.9%); in an instrumented pear run, keyframe fusion takes 250 ms without blocking the mapper.Source: C. Ablation Study · p. 8

VI. CONCLUSIONS

ArborSplat produces semantic orchard maps online and outperforms the evaluated semantic Gaussian SLAM systems on completed segments in accuracy and speed. Its remaining gap is held-out trunk semantics on full routes, which the authors identify as a target for future work.

VI. CONCLUSIONS

  • ArborSplat exceeds SGS-SLAM and GS3LAM on every completed segment by 0.23 to 0.50 training-view and 0.15 to 0.36 held-out mIoU while running 1.7 to 7.5 times faster.Source: §VI · p. 8
  • Its semantic point cloud is more accurate than naive transfer on all six sequences, and reduced-budget refinement raises training-view fruit IoU in all six paired runs and tree mIoU in four.Source: §VI · p. 8
  • On full routes, held-out trunk IoU stays below 0.07 on every sequence, versus 0.41 to 0.56 on training views.The authors plan to target novel-view semantics by revisiting past keyframes and closing loops.Source: §VI · p. 8
Loading 2609.26315v1…