Source-linked AI summary
Calibration and Comparative Analysis of Forward-Looking Sonar and 3D Sonar for Enhanced Underwater Object Recognition
Aditya Penumarti, Khanh Dong, Zi-Hao Zhang, Yongkyoon Park, Zhenqi Wu, Trung Dong, Shahriar Negahdaripour, Xiaomin Lin, Jane Shin
TL;DR
Underwater robots often rely on acoustic sensing because turbidity and low light degrade optical visibility, while traditional forward-looking sonar loses elevation information in its 2D projections. The paper aligns forward-looking sonar with 3D sonar through auto-calibration and uses the calibrated representation for denoising, improving calibration and segmentation outcomes.
Problem
Optical visibility degrades in turbid or deep environments, while traditional forward-looking sonar loses elevation information through 2D projection, complicating object recognition and spatial reasoning.
Method
The paper aligns an FLS and 3D sonar on the same ROV, extracting contours and optimizing their alignment using mutual information between the modalities.
Results
Auto-calibration improves overlap metrics by approximately 5% and distance metrics by 0.5px, while FLS-filtering improves Dice by 40% and IoU by 70% over the raw cloud.
Takeaways & Limitations
Calibrated multimodal denoising improves segmentation and object recognition relative to raw and traditional denoised point clouds, with larger advantages expected in highly turbid environments.
Takeaways & Limitations
Auto-calibration collapses elevation during feature matching, which can impair reconstruction or introduce erroneous noise into denoising.
Abstract
from arXiv · showhide
Sonars generate a significant amount of noise. With the advent of new technology capable of producing full 3D point clouds, the noise is amplified in sparse point clouds, making it challenging to recognize features for navigation, recognition, or reconstruction. To address this challenge, we propose using two different sonar modalities: one that produces a 2D intensity image and another that generates a 3D point cloud. By implementing auto-calibration, we can filter out noisy features between the modalities to enhance feature extraction. Experiments demonstrate that auto-calibration improves performance over manual calibration by 5% and that filtering enhances feature extraction by more than 40% relative to the raw point cloud. Code and datasets are given at https://theaprilab.org/fls-3d-calibrator
I. INTRODUCTION
Underwater robots rely on acoustic sensing because optical visibility degrades, but FLS loses elevation information and 3D sonar introduces density, processing, and cost trade-offs. The paper proposes jointly calibrating these modalities and releases supporting datasets and code.
- Acoustic sensing often serves as the primary modality for underwater robotic platforms when turbidity or depth degrades optical visibility.
- FLS projects 3D scenes onto 2D images, losing elevation information and complicating object recognition and spatial reasoning.
- 3D sonar provides explicit spatial coordinates but involves trade-offs in data density, processing power, and cost compared with FLS.
- The literature has a gap regarding systematic calibration and synergistic use of FLS and 3D sonar modalities.
- The proposed framework aligns FLS and 3D-sonar coordinate systems on a shared ROV platform using field data and camera-based ground-truth estimation.
- The paper contributes auto-calibration, calibrated denoising for recognition or reconstruction, and publicly released datasets for reproducibility and training.
II. PROBLEM FORMULATION
The problem formulation represents the unknown relationship between FLS imagery and 3D-sonar point clouds with transformation and projection parameters. An optimization minimizes a loss to produce an aligned representation for recognition.
- The FLS produces a flattened fan-transformed intensity image, while the 3D sonar generates a point cloud of acoustic returns in R3.
- The two sensors have an arbitrary extrinsic transform and unknown projection, represented by an unknown parameter vector θ.
- θ includes a y-axis translation, yaw angle, and scaling components defined by sensing-origin radius and azimuth.
- The framework defines a loss function over the point cloud and FLS image to optimize the unknown calibration parameters.
- The optimization produces an automatically calibrated, aligned representation for object recognition.
III. METHODOLOGY
The methodology projects 3D acoustic returns into the FLS image space rather than projecting image pixels into the point-cloud frame. Polar conversion and pixel mapping produce coordinates used for contour-based calibration.
- A. Projection: The method applies the 3D point cloud to the FLS image, flattening acoustic returns to emphasize object locations in the sensor frame.
- A. Projection: Transformed point-cloud coordinates are represented in the FLS image plane before conversion into polar coordinates.
- A. Projection: The polar representation uses range r and azimuth α to maintain compatibility with the FLS fan image.
- A. Projection: The polar coordinates are projected into FLS image coordinates using image dimensions, pixel radius, and the maximum acoustic-return radius.
- A. Projection: The resulting pixel coordinates define IP, which is used to extract contours for point-cloud calibration.
B. Contour Extraction
Contour extraction converts FLS and transformed point-cloud data into object boundaries for calibration. The pipeline thresholds FLS intensity and preprocesses the point-cloud image to create continuous contours.
- B. Contour Extraction: The method extracts thresholded convex-hull contours from two images, using a rectangular cinder block as the calibration target.
- B. Contour Extraction: The FLS image is converted to grayscale and thresholded to preserve high-intensity returns.
- B. Contour Extraction: Contours are traced from the binary mask using sequential raster scanning and 8-connectivity, then filtered by minimum area.
- B. Contour Extraction: The transformed point-cloud image is Gaussian-blurred, Canny-edge detected, morphologically closed, and filled to form a continuous mask.
C. Loss Functions
The calibration quality is evaluated with four loss functions comparing contours from the FLS and projected point cloud. These include geometric, statistical-dependence, overlap, and directional-coverage measures.
- Four loss functions measure calibration quality: Chamfer distance, mutual information, intersection-over-union, and coverage error.Their weighted sum is used for optimization.
- Chamfer distance measures dissimilarity between contour point sets using nearest-neighbor distances.Distances are computed in both directions and summed into the Chamfer loss.
- Mutual information measures statistical dependence between FLS and point-cloud contours, increasing when their projections are aligned.Its computation uses a joint intensity probability over both modalities and parameter set θ.
3) Intersection-Over-Union (IoU):
IoU measures overlap between the FLS ground-truth contour and predicted boundary, while coverage measures directional contour overlap. Both are converted into losses by subtracting them from 1.
- IoU measures the overlap between two bounding boxes or contours, ranging from 0% for no overlap to 100% for perfect overlap.Here, Ck,E is treated as ground truth and Ck,P as the predicted boundary.
- The IoU loss is defined as L(IOU) = 1−IOU(θ).
- Coverage error evaluates the fraction of one filled contour mask covered by the other.Unlike symmetric IoU, coverage is directional and is also converted into a loss by subtracting it from 1.
D. Optimization
The optimization combines weighted calibration losses and searches for parameters minimizing the resulting objective. Powell’s gradient-free routine runs per frame, with median solutions used to reject outliers.
- The combined calibration loss is a weighted sum of individual loss functions so that no single loss dominates optimization.Metric weights are denoted wm and assigned according to their significance.
- Weights are estimated by perturbing the unknown parameter set, measuring parameter-space distances, and correlating them with loss changes using Spearman correlation.The resulting weights are assigned to S(θ).
- Powell’s method performs gradient-free iterative one-dimensional searches along changing directions until convergence or the iteration limit.It searches for a local minimum in the parameter space.
- Per-frame optimization solutions are aggregated by their median because Powell’s method is sensitive to initialization and noise.The median rejects outlier frames caused by poor contours or spurious returns.
E. Denoising Scheme
The denoising scheme projects calibrated 3D points into the FLS image and retains points falling inside the FLS contour mask. The filtered cloud is then used to validate object-recognition performance.
- Calibrated point-cloud points are projected into FLS pixel coordinates and accepted or rejected according to the binary contour mask.The retained indices satisfy M(ui, vi) = 1.
- The resulting filtered point cloud is used to validate whether combining both sonar modalities benefits object-recognition performance.
- Two experiments validate calibration against ground truth and manual calibration, then assess denoising for improved feature recognition.
- Object-recognition comparisons include raw unfiltered clouds, statistical outlier removal, and FLS-filtered clouds.Table III reports performance on denoised data and cropped raw data.
A. Experimental Setup
Experiments used a BlueROV2 equipped with FLS, 3D sonar, and an RGB camera, evaluating calibration and denoising on cinder-block point clouds. Auto-calibration outperformed manual calibration across metrics and enabled efficient on-the-fly calibration without ground-truth measurements.
- Experimental platform: The BlueROV2 carried FLS, 3D sonar, and an RGB camera, with calibration data collected from four underwater cinder blocks in 4 s snapshots.A small subset was labeled for ground-truth confirmation.
- Feature-extraction evaluation: PointNet was trained on synthetic cinder-block clouds with occlusion and Gaussian noise, then evaluated on manually annotated raw and filtered clouds.The synthetic data modeled the blocks’ field arrangement and acoustic shadowing.
- Calibration validation: Auto-calibration outperformed manual calibration across all metrics, improving overlap metrics by approximately 5% and distance metrics by 0.5px.The metrics included Dice, IoU, Chamfer, and reprojection error.
- Calibration validation: Across 336 point cloud frames, auto-calibration produced around a 5% improvement in overlap metrics and a 0.5px distance improvement over manual calibration.Improvements on IoU and MI were marginal because edge loss and discretization heavily penalized these metrics.
- Calibration validation: Although optimization gains were minimal on the tested setup, auto-calibration enabled full calibration of both modalities without a ground-truth measurement for on-the-fly use.The paper attributes the limited gains to relatively clean data acquired in the tested setup.
C. Denoising Object Recognition
The study tested calibrated FLS-based denoising for PointNet point-wise segmentation against raw and SOR-filtered clouds. FLS filtering improved recognition metrics substantially, while elevation collapse remained a limitation of calibration.
- Recognition evaluation: Point-wise binary segmentation with PointNet evaluated whether calibration-enabled denoising improved downstream object recognition.The experiment tested benefit beyond geometric alignment.
- Recognition evaluation: The comparison used a raw cloud cropped to 6 meters, calibrated FLS filtering, and statistical outlier removal.SOR rejects points far from their neighbors as an additional geometric-denoising baseline.
- Recognition results: 40% relative improvement in Dice and 70% in IoU were achieved by FLS filtering over the raw cloud.The calibrated contour rejects noise lacking a corresponding 2D return.
- Recognition results: FLS filtering exceeded SOR by 9% in Dice and 16% in IoU.Noise close to objects can survive SOR but be rejected by the FLS contour.
- Limitations: Auto-calibration collapses elevation during feature matching, which can cause poorer calibration filters when 3D objects are insufficiently reconstructed or erroneous noise enters denoising.The paper proposes projecting the FLS image onto the 3D point cloud in future work to preserve elevation information.
- Recognition results: The confusion matrices and segmentation results compare raw, SOR-filtered, and calibrated FLS-3D sonar filtering before PointNet feature extraction.The reported confusion-matrix pattern is that FLS filtering primarily removes false positives dominating raw and SOR predictions.