PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Spatial Perception for Heterogeneous Robot Collaboration in Subterranean Mining Environments

Mario Alberto Valdes Saucedo, Akash Patel, Christoforos Kanellakis, George Nikolakopoulos

arXiv:2609.37419v1cs.RO

TL;DR

No single robot suits both long-range mine exploration and payload-intensive resource characterization, so the paper presents an onboard perception interface between an Explorer and an Inspector. It uses prompt-driven segmentation and geometric abstraction to produce inspection targets; field validation reports training-free operation across two environments and two mineralization types.

  • Problem

    The Explorer–Inspector handover needs compact, actionable mineral targets with location and viewing geometry, despite unknown mine conditions and no mine-specific model.

    Source: §I · pp. 1–2

  • Method

    An onboard pipeline segments natural-language-prompted minerals, grounds detections in LiDAR, clusters cross-view evidence into persistent regions, and converts them into oriented polygons in a 3D scene graph.

    Source: §I · p. 2

  • Results

    Field validation in a subterranean test facility and an active magnesite mine demonstrated robust, training-free operation on iron-vein and magnesite mineralization under realistic underground conditions.

    Source: §IV · p. 5, §V · p. 6

  • Takeaways & Limitations

    The shared map and scene graph provide a compact, actionable handover that can seed the Inspector’s close-range viewpoint planning.

    Source: §V · p. 6

Abstract

from arXiv · show

The autonomous extraction of deep mineral deposits in abandoned underground mines is fundamentally a multi-agent integration problem. No single platform simultaneously offers the mobility to traverse kilometers of degraded drifts and the sensing payload required to characterize an ore body. This article presents the onboard perception pipeline that bridges two heterogeneous agents within the PERSEPHONE autonomous mining mission. Which consist of a lightweight Explorer robot that maps an unknown mine and generates a 3D scene graph of inspection targets, by running a zero-shot, vision-language semantic segmentation stack that detects mineral deposits directly from natural-language prompts. The map and the graph are then handed to a second Inspector robot, which carries an advanced sensing payload and uses them to plan close-range inspection viewpoints. We detail the complete pipeline, with emphasis on the geometric abstraction that turns raw detections into actionable inspection targets, spanning per-view bounding-box generation, cross-view box merging, plane fitting, and polygon extraction, and we report an extensive field validation in a subterranean test facility and in an active magnesite mine, covering both iron-vein and magnesite mineralization under realistic, perceptually degraded conditions.

I. INTRODUCTION

The paper addresses how heterogeneous robots can hand off useful mineral findings when exploration and close-range characterization favor different platforms. It presents an onboard, prompt-driven perception pipeline that turns detections into oriented polygon targets for an Inspector and validates the approach in two environments and on two mineralization types.

I. INTRODUCTION

  • An Explorer–Inspector division of labor pairs agile, lightly instrumented exploration with slower, payload-rich close-range resource characterization.The Explorer discovers and coarsely localizes resources; the Inspector revisits promising regions for high-value sensing.Source: §I · p. 1
  • The handover must communicate mineral location, identity, and viewing geometry in a compact, real-time, onboard representation without a mine-specific model.The motivation is that a point-cloud map alone does not tell the Inspector where mineralization is or how to observe it.Source: §I · p. 1
  • The open-set pipeline uses zero-shot vision-language segmentation and LiDAR to detect and localize deposits from natural-language prompts without mine-specific training data.The prompts can specify different minerals, such as magnesite or iron vein.Source: §I · pp. 1–2
  • Cross-view association consolidates detections into persistent regions, which are plane-fitted and reduced to oriented polygons organized in a 3D scene graph.Each polygon encodes patch extent and a surface normal for inspection; the scene graph supports global reasoning and prioritization by accumulated mineralization.Source: §I · p. 2
  • Validation covers iron veins and magnesite in a subterranean test facility and an active mine under realistic underground conditions.The evaluation spans two qualitatively different environments and two mineralization types.Source: §I · p. 2

II. RELATED WORK

Prior subterranean robotics work emphasizes mapping and traversal, while open-vocabulary perception and scene understanding offer ways to recognize semantic targets. This paper focuses on packaging operator-defined minerals as compact, oriented targets that a second robot’s inspection planner can use.

II. RELATED WORK

  • Subterranean autonomy research has emphasized reaching and covering environments, leaving recognition of operator-defined resources and their handoff to a specialized robot as a complementary problem.Heterogeneous fleets are described as more effective than a single platform for covering subterranean environments.Source: §II · p. 2
  • CLIPSeg enables prompt-driven segmentation without task-specific training, while the paper examines its use on raw, low-light, dust-laden mine imagery and lifting predictions into 3D.Related work includes CLIP and LSeg as vision-language approaches to open-ended recognition and dense prediction.Source: §II · p. 2
  • The pipeline supplies oriented polygons with surface normals, closing the gap between surface-description inputs assumed by inspection planners and semantic findings from the Explorer.The Inspector builds on an adaptive planner that refines a prior view plan against the encountered surface.Source: §II · pp. 2–3

III. METHODOLOGY

The Explorer runs onboard mineral perception over its globally consistent LiDAR-inertial point-cloud map, then sends the Inspector both that map and oriented mineral polygons for close-range viewpoint planning.

III. METHODOLOGY

  • The Explorer incrementally builds a globally consistent mine point-cloud map using LiDAR-inertial odometry while running the perception pipeline onboard.The Explorer also runs the STAGE exploration planner.Source: §III · p. 3
  • The pipeline segments prompted minerals from RGB images, projects detections into the 3D map, clusters persistent regions, and fits a plane to emit an oriented polygon for each region.These operations are performed onboard during exploration.Source: §III · p. 3
  • The Inspector receives the accumulated point-cloud map and mineral polygons, then uses an adaptive planner to generate close-range viewpoints oriented along each polygon’s normal.The Inspector carries an advanced sensing payload such as a hyperspectral camera.Source: §III · p. 3

A. Zero-shot semantic and panoptic segmentation

The semantic front-end converts natural-language prompts into thresholded mineral masks, then separates distinct same-class deposits into instance labels by filtering small contours.

A. Zero-shot semantic and panoptic segmentation

  • CLIPSeg pairs a frozen CLIP backbone with a transformer decoder to produce a per-pixel confidence map for each natural-language prompt.The model applies a sigmoid to the prompt-specific logit map.Source: A. Zero-shot semantic and panoptic… · p. 3
  • A pixel is assigned to its most confident mineral class only when that class’s confidence exceeds threshold τ; otherwise it is labeled background.The thresholded argmax converts soft confidence maps into a hard multiclass mask and suppresses low-confidence responses.Source: A. Zero-shot semantic and panoptic… · p. 3
  • For each mineral class, external mask contours smaller than minimum area a_min are discarded, and each remaining contour receives a unique instance identifier.The area filter rejects small, flickering responses that would otherwise generate unstable detections.Source: A. Zero-shot semantic and panoptic… · p. 3

B. LiDAR projection and semantic point-cloud augmentation

The pipeline aligns image and LiDAR data, projects LiDAR points into the image, and transfers pixel semantics to retained points to create an augmented cloud.

B. LiDAR projection and semantic point-cloud augmentation

C. Cross-view clustering and persistent regions of interest

The global mapping node converts noisy per-frame detections into persistent mineral regions by filtering, placing points in the world frame, and associating overlapping detections across views.

C. Cross-view clustering and persistent regions of interest

  • The global mapping node uses augmented clouds and odometry to condense noisy, redundant single-frame detections into a stable set of mineral regions.Source: C. Cross-view clustering and… · p. 3
  • Per-instance DBSCAN clustering retains the largest cluster to reject outliers from masks spanning depth discontinuities.Such masks can project a few pixels onto distant geometry.Source: C. Cross-view clustering and… · pp. 3–4
  • Surviving points are transformed from the body frame into the global map frame using synchronized odometry, making detections from different viewpoints spatially comparable.Source: C. Cross-view clustering and… · p. 4
  • Same-class detections whose axis-aligned bounding boxes overlap are merged by concatenating their points; otherwise, a new box is created.Geometric overlap and class association let observations from multiple viewpoints accumulate into persistent regions of interest.Source: C. Cross-view clustering and… · p. 4

D. Plane fitting and polygon generation

Each persistent region becomes an oriented polygon that describes a finite mineral patch and its inspection orientation. The polygon is derived by fitting a plane to mineral points, projecting them onto it, and enclosing their projections in a minimum-area rectangle.

D. Plane fitting and polygon generation

  • RANSAC fits a dominant plane to mineral-labeled points accumulated inside a persistent bounding box, with the normal oriented toward observed free space.The local-planarity assumption applies to mineralization on a mine face; RANSAC handles off-surface points from projection noise and the rough wall.Source: D. Plane fitting and polygon generation · p. 4
  • The mineral points are projected onto the fitted plane, where a minimum-area bounding rectangle bounds the finite inspection target.The plane itself is infinite, so the rectangle localizes the observed patch.Source: D. Plane fitting and polygon generation · p. 4
  • The rectangle’s ordered corners are embedded back in 3D to form the polygon together with its plane normal and projection centroid.Source: D. Plane fitting and polygon generation · p. 4
  • Fitting the rectangle to the projected mineral points tightly covers their spatial support regardless of plane orientation, with size adapting to the observed patch rather than a hand-tuned parameter.Large elongated veins produce larger rectangles than compact pockets.Source: D. Plane fitting and polygon generation · p. 4
  • The polygon seeds the Inspector’s viewpoint planner, which stands off the centroid along the outward normal and refines coverage viewpoints for the robot footprint and tunnel geometry.The Explorer-to-Inspector message combines the polygons with the shared PCD map and is described as orders of magnitude smaller than the raw semantic cloud.Source: D. Plane fitting and polygon generation · p. 4

E. Graph-based organization of inspection targets

Inspection polygons are organized into a three-level scene graph that represents individual deposits, groups nearby deposits into areas, and places those areas under a mine root.

E. Graph-based organization of inspection targets

IV. EXPERIMENTAL VALIDATION

Field validation across an underground test facility and an active magnesite mine shows prompt-driven mineral detections can be condensed into oriented inspection targets and a compact Explorer-to-Inspector handover.

IV. EXPERIMENTAL VALIDATION

  • The pipeline was run onboard with calibrated RGB and LiDAR in the LTU SubT facility and an active magnesite mine, with deployment parameters held fixed across sites unless noted.LiDAR projection supplied depth and LiDAR-inertial estimation supplied odometry.Source: §IV · p. 5
  • Changing only the natural-language prompt transferred detection from thin iron veins to broader, diffuse magnesite patches without retraining or site-specific data collection.Both detections persisted despite rough surfaces, specular reflections, robot-light shadows, and airborne dust; iron-vein confidence maps were sharper than those for magnesite.Source: §IV · p. 5
  • The iron-vein and magnesite outputs show a Mine-rooted scene graph grouping mineral-area clusters and per-deposit boxes, with the same structure recovered across different drift geometries.The iron-vein figure also shows inspection polygons, surface normals, and per-point confidence; the magnesite figure shows corresponding scene-graph and target outputs.Source: §IV · p. 5, Figure 3 · p. 5, Figure 4 · p. 5
  • RANSAC plane fitting and minimum-area rectangle extraction produced oriented inspection polygons whose normals indicate the direction from which each deposit was observed.The complete handover pairs these polygons with the shared point-cloud map and is orders of magnitude smaller than the raw semantic cloud.Source: §IV · pp. 5–6
  • ∼38k raw mineral points were condensed into a few dozen persistent deposits and inspection polygons, roughly two orders of magnitude smaller than the raw semantic cloud.Compact deposits fit a single plane better; the largest deposits are less planar and are better described by multiple polygons.Source: Figure 5 · p. 6

V. CONCLUSIONS

The paper concludes that its onboard, open-set perception pipeline provides a geometric interface between heterogeneous mining robots. Field validation across two environments and mineralization types demonstrated robust, training-free operation under realistic underground conditions.

V. CONCLUSIONS

  • The pipeline grounds zero-shot, prompt-based mineral detections in LiDAR, clusters cross-view evidence into persistent regions, and converts them into oriented polygons organized in a 3D scene graph.The shared map and graph form a compact description for an Explorer to transmit to an Inspector for close-range viewpoint planning.Source: §V · p. 6
  • Field validation in a subterranean test facility and an active magnesite mine across iron-vein and magnesite mineralization demonstrated robust, training-free operation under realistic underground conditions.Source: §V · p. 6
Loading 2609.37419v1…