Source-linked AI summary
SG-AMP: Scene-Graph-Guided Active Perception and Semantics-Aware Motion Planning for Pepper Plants
Rohit Menon, Shiva Rudra Lolla, Niklas Mueller-Goldingen, Gokul Chenchani, Ribana Roscher, Maren Bennewitz
TL;DR
SG-AMP addresses incomplete RGB-D perception and motion planning around dense pepper plants, where uncertainty alone misses expected but unobserved attachments and uniform collision checking ignores plant-part consequences. It combines robust depth completion, persistent panoptic mapping, scene-graph-guided active sensing, and semantics-aware motion planning. On pepper data, it reports 55.27% semantic mIoU, 38.67% PQ, and 40.62 mm depth RMSE, while input-conditioned uncertainty improves NYUv2 NLL and AUSE.
Problem
Dense pepper-plant inspection requires representing occluding plant parts and planning near them despite incomplete or corrupted RGB-D measurements.
Source: §I · p. 1Method
SG-AMP integrates uncertainty-aware depth completion, persistent panoptic mapping, scene-graph hypotheses for missing attachments, active view selection, and semantics-aware motion planning.
Source: §I · p. 1Results
55.27% semantic mIoU, 38.67% PQ, and 40.62 mm depth RMSE are reported on pepper data, while input conditioning improves NYUv2 NLL and AUSE.
Source: §IV · p. 2Takeaways & Limitations
Missing pepper–peduncle attachments become explicit close-range inspection targets, while class-dependent costs protect critical plant organs and permit controlled foliage traversal.
Source: B. Panoptic Mapping and Plant Scene… · p. 2, C. Active View-Motion Planning · p. 2
Abstract
from arXiv · showhide
We present SG-AMP, integrating robust depth completion with input-conditioned uncertainty, persistent panoptic mapping, plant scene-graph reasoning, and semantics-aware active view-motion planning. Beyond inspecting uncertain observed regions, the scene graph explicitly hypothesizes unobserved pepper--peduncle attachments and directs close-range sensing toward them. Candidate views are selected according to expected information gain, while class-dependent motion costs distinguish protected peppers, peduncles, and stems from conditionally traversable foliage. On pepper data, the perception network achieves $55.27\%$ semantic mIoU, $38.67\%$ PQ, and $40.62\,\mathrm{mm}$ depth RMSE, while input-conditioned uncertainty improves NYUv2 NLL from $-1.6518$ to $-1.6925$ and AUSE from $0.0102$ to $0.0087$.
I. INTRODUCTION
SG-AMP addresses incomplete perception and unsafe uniform collision checking in dense pepper foliage by combining structured plant reasoning with active sensing and semantics-aware motion planning.
Motivation
- Dense pepper-plant inspection requires reasoning about peppers, peduncles, stems, and occluding leaves beyond fruit detection.RGB-D measurements may be incomplete or corrupted, and uncertainty alone does not identify expected but unobserved structures.Source: §I · p. 1
Approach
- SG-AMP integrates uncertainty-aware depth completion, persistent panoptic mapping, plant scene-graph reasoning, active view selection, and semantics-aware motion planning.Source: §I · p. 1
- Scene-graph hypotheses convert missing pepper–peduncle attachments into targeted inspection regions.The hypothesized attachment is represented as an inspection target rather than reconstructed geometry.Source: §I · p. 1
Illustration
- Figure 1 illustrates a fixed view, hypothesized occluded attachment, and arm-mounted close inspection of the highlighted region.Source: Figure 1 · p. 1
Approach
- Class-dependent motion costs protect peppers, peduncles, and stems while allowing controlled traversal of foliage.Source: §I · p. 1
II. RELATED WORK
Prior work improves fruit observation, semantic view selection, or persistent mapping, but does not jointly reason about pepper–peduncle–stem structure and semantic interaction costs.
Existing approaches
- Fruit-oriented active perception improves fruit observations without reasoning about the broader pepper–peduncle–stem structure.Source: §II · p. 1
- Semantic active-perception methods use semantic information for view selection without reasoning about instances.Source: §II · p. 1
- Fixed-camera panoptic plant mapping provides persistent 3-D representations without active sensing.Source: §II · p. 1
SG-AMP distinction
- SG-AMP uses plant structure to hypothesize missing attachments and plant-part semantics to selectively constrain sensor motion.Source: §II · p. 2
A. Robust RGB-D Perception
The perception module combines robust depth preprocessing, learned completion, input-conditioned uncertainty, and panoptic predictions for RGB-D plant mapping.
Depth completion
- Robust Fast Fill rejects locally inconsistent depth measurements using median/MAD filtering and restores supported measurements using spatial, RGB, and depth consistency.Source: A. Robust RGB-D Perception · p. 2
- Remaining missing regions are completed by a learned depth branch.Source: A. Robust RGB-D Perception · p. 2
Uncertainty
- Depth uncertainty is conditioned on measured, restored, or missing input status and continuous RFF support information.Source: A. Robust RGB-D Perception · p. 2
Panoptic perception
- The panoptic branch provides semantic evidence and instance predictions for 3-D fusion.Source: A. Robust RGB-D Perception · p. 2
B. Panoptic Mapping and Plant Scene Graph
SG-AMP maintains a persistent multi-resolution panoptic map and uses plant scene-graph relations to target expected but unobserved attachments for active confirmation.
Panoptic mapping
- Completed depth, semantic evidence, and instance masks are fused into a persistent multi-resolution panoptic map.Source: B. Panoptic Mapping and Plant Scene… · p. 2
- Depth updates are uncertainty-weighted, while semantic evidence accumulates across observations.Source: B. Panoptic Mapping and Plant Scene… · p. 2
- Peppers, peduncles, and stems retain persistent identities, whereas leaves and background classes remain semantic representations.Source: B. Panoptic Mapping and Plant Scene… · p. 2
Plant scene graph
- A pepper lacking an observed peduncle attachment induces a hypothesis toward a nearby stem for close-range inspection.Source: B. Panoptic Mapping and Plant Scene… · p. 2
- Hypothesized regions are inspection targets, not reconstructed geometry, and are confirmed, updated, or removed through active sensing.Source: B. Panoptic Mapping and Plant Scene… · p. 2
C. Active View-Motion Planning
SG-AMP selects reachable camera views for close-range inspection by balancing expected information gain against motion and semantic interaction costs.
Candidate view generation
- Candidate camera poses are sampled around incomplete plant parts and missing-peduncle hypotheses for close-range inspection.Ray casting estimates visibility and expected reductions in geometric, semantic, and association uncertainty.Source: C. Active View-Motion Planning · p. 2
Semantics-aware motion costs
- Peppers, peduncles, and stems receive high motion penalties, while leaves receive a lower finite cost and may be traversed when visibility requires it.The semantic cost is evaluated from mapped plant-part class distributions while retaining semantic uncertainty.Source: C. Active View-Motion Planning · p. 2
Evaluation context
- Table I is identified as reporting depth-restoration and uncertainty ablations on NYUv2.Source: Table I · p. 2
- Table II is identified as reporting panoptic RGB-D perception on the pepper-plant dataset.Source: Table II · p. 2
View selection
- Among kinematically reachable views with ESDF-feasible arm trajectories, selection trades expected information gain against motion and semantics-aware interaction costs.The weights λm and λs control the two penalties, and the map and scene graph are updated after execution.Source: C. Active View-Motion Planning · p. 2
IV. PRELIMINARY RESULTS
Preliminary experiments report improved depth restoration and uncertainty estimation on NYUv2, alongside panoptic perception results on pepper plants.
Pepper-plant perception
- 55.27% semantic mIoU and 38.67% PQ are achieved by the panoptic RGB-D model on the pepper-plant dataset.The substantially lower peduncle PQ relative to semantic IoU motivates targeted close-range inspection of thin, occluded attachments.Source: §IV · p. 2
Depth and uncertainty
- RFF improves denoising and completion under corrupted sparse-depth input on NYUv2.Input conditioning also improves NLL and AUSE over a shared uncertainty prior.Source: §IV · p. 2
V. PLANNED EVALUATION
The planned evaluation compares SG-AMP with passive, semantic, scene-graph, and complete-system baselines under matched initialization and sensing budgets.
Baselines
- The study will compare passive panoptic mapping, semantic next-best-view selection, scene-graph-guided sensing with conventional collision avoidance, and complete SG-AMP.All methods are evaluated under identical initialization and sensing budgets.Source: §V · p. 2
Evaluation metrics
- Evaluation will jointly quantify reconstruction, peduncle detection, sensing motion, and class-dependent plant interaction risk.Source: §V · p. 2
Ablations
- Ablations will isolate structural-hypothesis and semantics-aware-motion-cost contributions and compare against a deep deterministic uncertainty RGB-D baseline.Source: §V · p. 2
VI. CONCLUSION
SG-AMP combines robust RGB-D perception, uncertainty-aware panoptic mapping, scene-graph-guided active sensing, and semantics-aware motion planning for pepper plants.
Conclusion
- Missing pepper–peduncle attachments provide explicit targets for additional observation beyond uncertain observed regions.Source: §VI · p. 2
- Class-dependent motion costs distinguish protected plant structures from conditionally traversable foliage.Source: §VI · p. 2
- Preliminary results demonstrate improved depth restoration and uncertainty estimation together with panoptic perception of pepper plants.Source: §VI · p. 2