PaperlayerView exact revision on arXiv

Source-linked AI summary

Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation

Borja Bovcon, Rok Mandeljc, Janez Perš, Matej Kristan

arXiv:1802.07956v1cs.ROcs.CV

TL;DR

Marine USV obstacle detection must handle limited sensing, difficult water imagery, and boat motion. The paper combines IMU-assisted semantic segmentation with stereo verification and evaluates it on a new synchronized multi-sensor dataset. The resulting system substantially improves reported detection measures while remaining real-time, although stereo processing reduces speed and some visually consistent or very small obstacles remain problematic.

  • Problem

    Existing marine obstacle detectors are limited by difficult water imagery and boat motion, while standard stereo methods struggle with reflections, changing surfaces, weak texture, and partially submerged obstacles.

  • Method

    The paper extends a semantic-segmentation graphical model with IMU roll and pitch to project the horizon and adjust priors, then applies stereo verification to candidate detections.

  • Results

    The proposed approach significantly outperforms the state-of-the-art, including 29.8% better sea-edge estimation and 45.0% higher F-score for ISSM versus SSM.

  • Takeaways & Limitations

    The system improves marine water-edge and obstacle detection across challenging conditions while retaining real-time operation.

  • Takeaways & Limitations

    Stereo extension reduces processing speed by approximately 50%, and verification can retain visually consistent false positives or remove a true detection when a buoy appears very small.

Abstract

from arXiv · show

A new obstacle detection algorithm for unmanned surface vehicles (USVs) is presented. A state-of-the-art graphical model for semantic segmentation is extended to incorporate boat pitch and roll measurements from the on-board inertial measurement unit (IMU), and a stereo verification algorithm that consolidates tentative detections obtained from the segmentation is proposed. The IMU readings are used to estimate the location of horizon line in the image, which automatically adjusts the priors in the probabilistic semantic segmentation model. We derive the equations for projecting the horizon into images, propose an efficient optimization algorithm for the extended graphical model, and offer a practical IMU-camera-USV calibration procedure. Using an USV equipped with multiple synchronized sensors, we captured a new challenging multi-modal dataset, and annotated its images with water edge and obstacles. Experimental results show that the proposed algorithm significantly outperforms the state of the art, with nearly 30% improvement in water-edge detection accuracy, an over 21% reduction of false positive rate, an almost 60% reduction of false negative rate, and an over 65% increase of true positive rate, while its Matlab implementation runs in real-time.

1. Introduction

Small USVs benefit from lightweight camera sensing, but marine obstacle detection remains difficult because visual conditions and boat motion undermine existing methods. The paper extends semantic segmentation with IMU measurements and stereo verification, and evaluates it on a new synchronized multi-sensor dataset.

  • Motivation: Small USVs have limited sensor payload and power, making lightweight, passive cameras a practical sensing option.Cameras avoid moving parts and potential interference risks associated with active sensors.
  • Motivation: Rapidly changing water surfaces, reflections, weak texture, and partially submerged obstacles limit standard stereo-vision methods.The prior semantic-segmentation approach also fails during substantial boat rolling and pitching.
  • Contributions: The proposed detector incorporates onboard IMU roll and pitch measurements into semantic segmentation and verifies detections with stereo vision.IMU data projects the horizon into the image and adjusts segmentation priors, while epipolar constraints and template matching support verification.
  • Contributions: The paper introduces a challenging dataset with synchronized stereo, IMU, and GPS streams plus water-edge and obstacle annotations.The dataset is described as the largest of its kind for marine obstacle detection.
  • Contributions: Experiments report that the proposed approach outperforms the state-of-the-art method in both sea-edge estimation and obstacle detection accuracy.The evaluation uses the newly collected multi-sensor dataset.

2. Related work

Prior marine obstacle-detection methods use stereo, radar, lidar, horizon estimation, or semantic segmentation, but each faces domain-specific limitations. The paper builds on semantic segmentation while addressing motion-related failures with IMU information and stereo verification.

  • Marine obstacle detection: Ground-vehicle obstacle detectors often rely on ground-plane estimation, which does not transfer readily to aquatic USV environments.Marine systems commonly use radar, sonar, or lidar, but these sensors add detection, size, or power-consumption challenges.
  • Horizon estimation: Horizon-based monocular methods can fail when the sea edge is not straight or when the sea-sky boundary is blurred.A straight-line horizon assumption is especially problematic in coastal waters and marinas.
  • Semantic segmentation: The prior semantic-segmentation model partitions marine images into sky, ground or haze, and water using Gaussian components and a Markov random field.It detects protruding and floating obstacles without assuming a straight water edge and runs in real time.
  • Proposed direction: The proposed work extends semantic segmentation with continuously updated IMU-informed hyper-priors to reduce visual ambiguity and improve water-edge and obstacle detection.Its stereo extension then verifies candidate detections using cross-view information.

3. Augmented semantic segmentation model

The augmented semantic segmentation model incorporates IMU-derived horizon information into pixel priors and Gaussian hyper-priors, adapting semantic regions to USV motion and shoreline geometry. It then extracts water regions and enclosed non-water blobs as potential obstacles.

  • Model formulation: The model represents each pixel with position and RGB features in a four-component mixture model for three semantic regions plus an outlier component.The Gaussian components correspond to semantic elements, while a uniform component models outliers.
  • Motion-dependent hyper-priors: Roll and pitch modify Gaussian hyper-priors so component positions and orientations follow the changing projected horizon.Horizon-dependent means use learned vertical displacements, while the middle component covariance is rotated according to the horizon slope.
  • Optimization: The extended graphical model optimizes priors and posteriors with an EM-like procedure that smooths neighboring distributions and incorporates horizon-conditioned probabilities.The alternating E- and M-steps require only a few iterations to converge in practice.
  • Horizon-conditioned priors: IMU-derived horizon parameters condition per-pixel class priors, making water impossible above the horizon and sky impossible below it.The conditional priors for water and sky are blurred with a small-variance Gaussian to accommodate horizon-estimation noise.
  • Horizon projection: The horizon is projected from distant water-surface points using camera calibration and USV-to-camera rotation, with projection governed by camera-to-plane rotation rather than camera height.The calibration matrix is estimated during the calibration process.
  • Obstacle extraction: The resulting water mask is reduced to its largest connected region, whose enclosed non-water blobs become potential obstacles after non-maximum suppression.The largest connected outer edge of the obstacle map defines the water edge.

4. Stereo verification for improved detection

Stereo verification consolidates tentative obstacle detections from independently segmented left and right images using epipolar constraints and normalized cross-correlation (NCC). It addresses false positives while revisiting unmatched detections to retain obstacles visible in only one image.

  • Detection consolidation: Stereo verification applies the segmentation model independently to the left and right cameras, then matches detections using epipolar constraints and NCC.The complete procedure also revisits unpaired detections with brute-force template matching.
  • Epipolar verification: An obstacle detected in the left image is expected near its corresponding epipolar line in the right image, allowing a bounded correspondence search.The vicinity is set using the obstacle bounding-box diagonal to accommodate imperfect segmentation and calibration.
  • Template matching: NCC matches an extracted obstacle template against an enlarged search region across the three color channels, accepting matches above θncc.The channel-specific response maps are averaged into one response map before thresholding.
  • Unpaired detections: Unmatched tentative detections are revisited in the opposite image, so objects detected in only one camera can still pass NCC verification.Detections that fail verification are discarded, and the resulting system is denoted ISSMS.
  • Algorithm output: ISSMS outputs an obstacle image map, water edge, detected objects, and updated model parameters after stereo matching and template verification.Its algorithm takes rectified stereo images, horizon parameters, and the previous time-step model as inputs.

5. The camera-IMU calibration

The camera-IMU calibration estimates the rotations needed to project the IMU-derived horizon into the camera. It combines a flat-ground setup, IMU readings, and stereo point-cloud plane fitting.

  • Calibration transforms: Horizon projection depends on IMU-to-world and camera/IMU-to-USV rotations, with the latter calibration matrices estimated from the sensor configuration.The IMU rotation is constructed at each time step from IMU readings, while fixed sensor-to-USV rotations require calibration.
  • Flat-ground setup: Calibration places the USV on flat ground so the world and USV coordinate systems are aligned, allowing one rotation to be obtained directly from the IMU.This setup provides the reference geometry for estimating the remaining camera rotation.
  • Ground-plane estimation: The camera rotation can be estimated from a floor-plane orientation obtained either from a planar marker homography or from stereo point-cloud fitting.The implemented procedure uses the stereo point cloud and RANSAC to fit points within a predefined distance threshold.
  • Rotation recovery: The camera-to-USV rotation is computed from the fitted plane normal, with rotation around the camera Y-axis set to zero because that axis points along the heading direction.The plane normal directly provides rotations around the X- and Z-axes.

6. Multimodal marine obstacle detection dataset

The Modd 2 dataset was created to provide annotated multimodal USV obstacle-detection data under varied navigation and environmental conditions. It contains stereo video, IMU-related sensing, water-edge and obstacle annotations, and sequences recorded in coastal waters.

  • Dataset scale: Modd 2 contains 28 video sequences totaling 11,675 frames at 1278 × 958 resolution.The dataset was created because realistic publicly available USV data combining annotated stereo video and IMU data were lacking.
  • Acquisition platform: The data were collected over approximately 15 months in the Gulf of Koper, Slovenia, using a USV equipped with a compass, IMU, and synchronized stereo cameras.The stereo system has a 132.1° field of view and is mounted approximately 0.7 m above the water surface.
  • Navigation scenarios: Manual acquisition simulated hazardous navigation by including trajectories toward obstacles and passages near obstacles.These scenarios represent both direct collision courses and close-proximity encounters.
  • Annotations: Each frame was manually annotated and expert-verified with a water-edge polygon, obstacle bounding boxes, and optional horizon annotations.Obstacles are divided into large objects straddling the sea edge and small objects located fully below it.
  • Environmental variation: The dataset includes varied conditions such as sudden motion, rolling and pitching, sun glitter, and environmental reflections.Twenty-five of 28 sequences contain at least one obstacle, and the average obstacle count is 0.614 per frame with standard deviation 0.945.

7. Experiments

Experiments evaluate the Matlab ISSMS prototype against state-of-the-art methods in monocular and stereo settings, including ablations, speed analysis, and extreme-condition testing. The implementation uses the SSM parameters and RGB segmentation, with additional horizon-related and stereo-verification settings.

  • Implementation: The ISSMS prototype is implemented in Matlab and evaluated on a desktop with an Intel Core i7-7700 3.6 GHz CPU.The authors state that the code is planned for public release.
  • Evaluation design: The experiments compare monocular performance, conduct an ablation study, evaluate stereo performance, analyze speed, and test extreme conditions and failure cases.These evaluations are organized across Sections 7.2–7.7.
  • Model settings: Segmentation uses RGB inputs and the same SSM parameters as the prior method, while hyper-priors are trained on the same sequences with additional horizon annotations.The horizon projection and stereo-verification parameters are specified separately for the implementation.
  • Horizon projection: Horizon-point generation keeps fitted-line error low below 56° and uses αh = 40° in the experiments.The reported error is the normalized RMSE of the fitted horizon line across generated point angles.
  • Calibration parameters: Camera-IMU calibration uses a 10 m obstacle-free region, a 1 m RANSAC point-to-plane threshold, and up to 1000 iterations.The 10 m region determines the distance threshold used for plane estimation in the marina parking lot.
  • Stereo parameters: Stereo verification uses search-region scale factors θS1 = 1.2 and θS2 = 3.0 with a conservative NCC threshold θncc = 0.95.These values were selected experimentally on a separate training set, and the NCC threshold was not fine tuned.

7.2. Comparison to state of the art: monocular setup

The IMU-assisted semantic segmentation method improves monocular sea-edge estimation and obstacle detection over the original SSM, including under blurred-horizon conditions. Ablation results indicate that the full combination of model modifications performs best overall.

  • 66.9% reduction in false negatives and 61.3% increase in true positives improve obstacle detection with ISSM.The corresponding F-score improvement is 44.9%.
  • ISSM correctly segments images in harsh conditions where the horizon is blurred, whereas SSM often overestimates water extent.The overestimation can cause obstacles to be mistaken for water.
  • Ablation study: ISSMM3 improves sea-edge estimation by 31.0% and F-score by 44.0% compared with the original SSM.Among the ablation variants, ISSMM3 performs best on both reported measures before the full combination is applied.
  • Ablation study: The full ISSM combination achieves the best overall performance among the tested variants.Each described variant outperforms the original SSM in sea-edge estimation error and F-score.

7.4. Comparison to state of the art: stereo verification

Adding stereo verification to the segmentation pipeline substantially reduces false positives and improves F-score relative to both monocular baselines. The verification removes detections associated with several visual artifacts.

  • The stereo comparison evaluates ISSMS against ISSM and SSMS against SSM for fair monocular and stereo baselines.
  • 95.2% reduction in false positives and 46.5% F-score improvement are obtained when comparing ISSMS with ISSM.Across the reported comparisons, stereo verification improves F-score by approximately 41% on average.
  • 90.9% reduction in false positives and 35.7% F-score improvement are obtained when comparing SSMS with SSM.
  • Stereo verification removes false positives caused by sea foam, small sun glitter, and sun flares on the camera lens.

7.5. Computational performance analysis

IMU processing preserves the original monocular processing speed, while stereo processing is slower but remains real-time for the target USV. Performance varies with the number of obstacles requiring verification and under extreme conditions.

  • Approximately 30 fps is maintained when IMU information is added to the original SSM.The speed is maintained while sea-edge estimation and obstacle-detection performance improve.
  • Stereo segmentation is approximately 50% slower than monocular segmentation because it processes two images.Obstacle verification adds a further reduction that increases with the number of detected obstacles.
  • The stereo algorithm remains real-time on the target USV because its cameras support no more than 10 frames per second.The stereo setup improves obstacle-detection quality by 46.5%.
  • Extreme conditions: Abrupt USV motion can make the estimated horizon lag behind its true position because filtered IMU readings do not properly reflect pose.This may reduce the accuracy of the conditional priors and affect image segmentation.
  • Extreme conditions: Under water reflections, ISSM improves sea-edge estimation by 46.8% and F-score by 42.9% over SSM.The reported reflection conditions include boats, land, piers, and buoys.
  • Extreme conditions: Under sun glitter, ISSM improves sea-edge estimation by as much as 4.3% compared with SSM.Stereo verification removes phantom detections caused by glitter.

7.7. Failure cases

Failure cases arise from visual ambiguity in segmentation and from stereo verification errors. The reported examples include reflections, sun glitter, sea foam, visually consistent false obstacles, and missed small buoy detections.

  • Sun glitter and opaque reflections cause water-region mislabeling, while a visually similar pier can be labeled as water.The water edge is estimated conservatively to prevent potentially dangerous false negatives near the boat.
  • Stereo verification can retain false positives that are consistent across views, including a pier detection caused by visual similarity.The method removes glitter detections but cannot remove every visually consistent false obstacle.
  • Stereo verification incorrectly removes a true buoy detection when the buoy appears very small at the observed distance.

8. Conclusion

The proposed ISSM model extends semantic segmentation with synchronized sensing and stereo verification, improving detection performance while retaining real-time operation. Evaluation uses a new multi-sensor USV dataset and reports gains in challenging conditions, alongside remaining implementation limitations.

  • Dataset: The study introduces a challenging synchronized video-and-IMU dataset with water-edge and obstacle annotations for USV evaluation.The dataset is described as the largest multi-sensor USV dataset and is intended for public release.
  • Experimental results: 29.8% improvement in sea-edge estimation and 45.0% improvement in F-score are reported for ISSM over SSM.These results come from experiments comparing the monocular ISSM with the state-of-the-art SSM baseline.
  • Experimental results: 54.2% improvement over SSM is reported for ISSMS with added stereo verification.The reported comparison concerns the stereo extension of the proposed method.
  • Extreme conditions: Over 68% improvement is reported when stereo verification is added to ISSM, while ISSMS exceeds SSMS by approximately 24%.These gains are reported for extreme conditions and comparisons involving stereo verification.
  • Runtime: The stereo extension reduces processing speed by approximately 50%, but remains above the camera frame rate and therefore real-time.ISSM itself does not incur a significant processing-speed drop relative to SSM.
  • Limitations and future work: The reported evaluation was performed offline in Matlab, and future work plans onboard reimplementation, broader dataset evaluation, and stronger learned visual features.The current method uses simple visual features and has not yet been reimplemented on the USV onboard system.
Loading 1802.07956v1…