Source-linked AI summary
SG-AMP: Scene-Graph-Guided Active Perception and Semantics-Aware Motion Planning for Pepper Plants
Rohit Menon, Shiva Rudra Lolla, Niklas Mueller-Goldingen, Gokul Chenchani, Ribana Roscher, Maren Bennewitz
TL;DR
SG-AMP addresses incomplete RGB-D perception and motion planning around dense pepper plants, where uncertainty alone misses expected but unobserved attachments and uniform collision checking ignores plant-part consequences. It combines robust depth completion, persistent panoptic mapping, scene-graph-guided active sensing, and semantics-aware motion planning. On pepper data, it reports 55.27% semantic mIoU, 38.67% PQ, and 40.62 mm depth RMSE, while input-conditioned uncertainty improves NYUv2 NLL and AUSE.
Problem
Dense pepper-plant inspection requires representing occluding plant parts and planning near them despite incomplete or corrupted RGB-D measurements.
Method
SG-AMP integrates uncertainty-aware depth completion, persistent panoptic mapping, scene-graph hypotheses for missing attachments, active view selection, and semantics-aware motion planning.
Results
55.27% semantic mIoU, 38.67% PQ, and 40.62 mm depth RMSE are reported on pepper data, while input conditioning improves NYUv2 NLL and AUSE.
Takeaways & Limitations
Missing pepper–peduncle attachments become explicit close-range inspection targets, while class-dependent costs protect critical plant organs and permit controlled foliage traversal.
Abstract
from arXiv · showhide
We present SG-AMP, integrating robust depth completion with input-conditioned uncertainty, persistent panoptic mapping, plant scene-graph reasoning, and semantics-aware active view-motion planning. Beyond inspecting uncertain observed regions, the scene graph explicitly hypothesizes unobserved pepper--peduncle attachments and directs close-range sensing toward them. Candidate views are selected according to expected information gain, while class-dependent motion costs distinguish protected peppers, peduncles, and stems from conditionally traversable foliage. On pepper data, the perception network achieves $55.27\%$ semantic mIoU, $38.67\%$ PQ, and $40.62\,\mathrm{mm}$ depth RMSE, while input-conditioned uncertainty improves NYUv2 NLL from $-1.6518$ to $-1.6925$ and AUSE from $0.0102$ to $0.0087$.
I. INTRODUCTION
SG-AMP addresses incomplete perception and unsafe uniform collision checking in dense pepper foliage by combining structured plant reasoning with active sensing and semantics-aware motion planning.
- Motivation: Dense pepper-plant inspection requires reasoning about peppers, peduncles, stems, and occluding leaves beyond fruit detection.RGB-D measurements may be incomplete or corrupted, and uncertainty alone does not identify expected but unobserved structures.
- Approach: SG-AMP integrates uncertainty-aware depth completion, persistent panoptic mapping, plant scene-graph reasoning, active view selection, and semantics-aware motion planning.
- Approach: Scene-graph hypotheses convert missing pepper–peduncle attachments into targeted inspection regions.The hypothesized attachment is represented as an inspection target rather than reconstructed geometry.
- Illustration: Figure 1 illustrates a fixed view, hypothesized occluded attachment, and arm-mounted close inspection of the highlighted region.
- Approach: Class-dependent motion costs protect peppers, peduncles, and stems while allowing controlled traversal of foliage.
II. RELATED WORK
Prior work improves fruit observation, semantic view selection, or persistent mapping, but does not jointly reason about pepper–peduncle–stem structure and semantic interaction costs.
- Existing approaches: Fruit-oriented active perception improves fruit observations without reasoning about the broader pepper–peduncle–stem structure.
- Existing approaches: Semantic active-perception methods use semantic information for view selection without reasoning about instances.
- Existing approaches: Fixed-camera panoptic plant mapping provides persistent 3-D representations without active sensing.
- SG-AMP distinction: SG-AMP uses plant structure to hypothesize missing attachments and plant-part semantics to selectively constrain sensor motion.
A. Robust RGB-D Perception
The perception module combines robust depth preprocessing, learned completion, input-conditioned uncertainty, and panoptic predictions for RGB-D plant mapping.
- Depth completion: Robust Fast Fill rejects locally inconsistent depth measurements using median/MAD filtering and restores supported measurements using spatial, RGB, and depth consistency.
- Depth completion: Remaining missing regions are completed by a learned depth branch.
- Uncertainty: Depth uncertainty is conditioned on measured, restored, or missing input status and continuous RFF support information.
- Panoptic perception: The panoptic branch provides semantic evidence and instance predictions for 3-D fusion.
B. Panoptic Mapping and Plant Scene Graph
SG-AMP maintains a persistent multi-resolution panoptic map and uses plant scene-graph relations to target expected but unobserved attachments for active confirmation.
- Panoptic mapping: Completed depth, semantic evidence, and instance masks are fused into a persistent multi-resolution panoptic map.
- Panoptic mapping: Depth updates are uncertainty-weighted, while semantic evidence accumulates across observations.
- Panoptic mapping: Peppers, peduncles, and stems retain persistent identities, whereas leaves and background classes remain semantic representations.
- Plant scene graph: A pepper lacking an observed peduncle attachment induces a hypothesis toward a nearby stem for close-range inspection.
- Plant scene graph: Hypothesized regions are inspection targets, not reconstructed geometry, and are confirmed, updated, or removed through active sensing.
C. Active View-Motion Planning
SG-AMP selects reachable camera views for close-range inspection by balancing expected information gain against motion and semantic interaction costs.
- Candidate view generation: Candidate camera poses are sampled around incomplete plant parts and missing-peduncle hypotheses for close-range inspection.Ray casting estimates visibility and expected reductions in geometric, semantic, and association uncertainty.
- Semantics-aware motion costs: Peppers, peduncles, and stems receive high motion penalties, while leaves receive a lower finite cost and may be traversed when visibility requires it.The semantic cost is evaluated from mapped plant-part class distributions while retaining semantic uncertainty.
- Evaluation context: Table I is identified as reporting depth-restoration and uncertainty ablations on NYUv2.
- Evaluation context: Table II is identified as reporting panoptic RGB-D perception on the pepper-plant dataset.
- View selection: Among kinematically reachable views with ESDF-feasible arm trajectories, selection trades expected information gain against motion and semantics-aware interaction costs.The weights λm and λs control the two penalties, and the map and scene graph are updated after execution.
IV. PRELIMINARY RESULTS
Preliminary experiments report improved depth restoration and uncertainty estimation on NYUv2, alongside panoptic perception results on pepper plants.
- Pepper-plant perception: 55.27% semantic mIoU and 38.67% PQ are achieved by the panoptic RGB-D model on the pepper-plant dataset.The substantially lower peduncle PQ relative to semantic IoU motivates targeted close-range inspection of thin, occluded attachments.
- Depth and uncertainty: RFF improves denoising and completion under corrupted sparse-depth input on NYUv2.Input conditioning also improves NLL and AUSE over a shared uncertainty prior.
V. PLANNED EVALUATION
The planned evaluation compares SG-AMP with passive, semantic, scene-graph, and complete-system baselines under matched initialization and sensing budgets.
- Baselines: The study will compare passive panoptic mapping, semantic next-best-view selection, scene-graph-guided sensing with conventional collision avoidance, and complete SG-AMP.All methods are evaluated under identical initialization and sensing budgets.
- Evaluation metrics: Evaluation will jointly quantify reconstruction, peduncle detection, sensing motion, and class-dependent plant interaction risk.
- Ablations: Ablations will isolate structural-hypothesis and semantics-aware-motion-cost contributions and compare against a deep deterministic uncertainty RGB-D baseline.
VI. CONCLUSION
SG-AMP combines robust RGB-D perception, uncertainty-aware panoptic mapping, scene-graph-guided active sensing, and semantics-aware motion planning for pepper plants.
- Conclusion: Missing pepper–peduncle attachments provide explicit targets for additional observation beyond uncertain observed regions.
- Conclusion: Class-dependent motion costs distinguish protected plant structures from conditionally traversable foliage.
- Conclusion: Preliminary results demonstrate improved depth restoration and uncertainty estimation together with panoptic perception of pepper plants.